How to escape `*` character in pipe variables?

[Łukasz Świerżewski]

I'm trying to use `atlassian/aws-s3-deploy` with `EXTRA_ARGS` pipe variable like: `--exclude * --include images/*` but single `*` character in `--exclude` argument is replaced with all files listed in the current category separated by a space.

Final command run by pipe should look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude * --include images/*

But in debug mode I see:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude app.php css images js --include 'images/*'

As you can see, `*` character was replaced with file list which break command to work.

If I quote (or double quote) `EXTRA_ARGS` the `*` is not replaced but executed command look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ '--exclude * --include images/*'

which also break command.

If I quote (or double quote) only `*` is not replaced and executed command look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude '*' --include images/*

but then in debug mode I can see:

020-01-02 00:01:27,529 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/'*'

so it use wrong value with filter (not remove quotes) and results with not match.

Is there any way to escape `*` that it will not be replaced?

[Łukasz Świerżewski · Accepted solution]

Thanks for reply.

I tried your suggestion, but `atlassian/aws-s3-deploy` doesn't support `Advanced techniques for writing pipes

Finally, I resolve problem using bash equals-separated options notation writing `--exclude=*` instead of `--exclude *` (`=` instead of space) which doesn't extend glob and results command like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude=* --include=images/*

[ktomk]

Ah nice, I was not aware that aws cli supports this mode but actually checked it in their docs which did not outline it at the command, I think they cover it in the general description, yeah found it:

Optionally, you can optionally separate the parameter name from the value with an equals sign (=) instead of a space. This is typically necessary only if the value of the parameter starts with a hyphen.

From: Specifying Parameter Values for the AWS CLI

Another last resort might be to provide a json template as many aws cli commands support those, but would be more work. Just if someone stumbles of it here. It is in the same link: Using JSON for Parameters .

[Szabolcs Fodor]

Great! Thanks!
It seems the same trick seems to work with the Azure CLI pipe (atlassian/azure-cli-run:1.1.0) as well. E.g.:

az cdn endpoint purge --resource-group=rg --profile-name=profile --name=endpoint --content-paths=/*

[ktomk]

Looks like the problem is with glob expansions with the EXTRA_ARGS parameter of the pipe.

Try with a different notation in the pipelines YAML:

variables:
  EXTRA_ARGS: ['--exclude', '*', ...]

This is in reference to: Advanced techniques for writing pipes 

If the Pipe you use follows this reference (I would guess so as it looks like you're using the one offered by Atlassian here) , this might be more explicit and better to control. But you need to try as I don't know for sure.

Additionally I remember a similar issue not with glob expansion but with variable substitution and EXTRA_ARGS: Way to use "---include-from" in rsync deploy - In the end, the variable substitution only worked by not putting the variable name into curly brackets:

EXTRA_ARGS: '--include-from=$BITBUCKET_CLONE_DIR/deploy-include.txt --exclude=*'

Note that this is the rsync utility not the aws utility, but also a pipe from Atlassian.

So why not give the different notation a try and post the outcome. This might shed some more light.

This should also help to find out if there is a general problem with the pipe regarding --exclude / --include patterns (vendor docs about the patterns) which should then be reported against the concrete pipe repository (atlassian/aws-s3-deploy).

[Łukasz Świerżewski]

I tried and asterisk is not replaced but invlaid value is passed to aws command:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude \* --include images/*

and filtering faild with results:

2020-01-02 06:34:16,533 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/\*

[Pramodh M]

Try \*