Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to escape `*` character in pipe variables?

I'm trying to use `atlassian/aws-s3-deploy` with `EXTRA_ARGS` pipe variable like: `--exclude * --include images/*` but single `*` character in `--exclude` argument is replaced with all files listed in the current category separated by a space.

Final command run by pipe should look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude * --include images/*

But in debug mode I see:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude app.php css images js --include 'images/*'

As you can see, `*` character was replaced with file list which break command to work.

If I quote (or double quote) `EXTRA_ARGS` the `*` is not replaced but executed command look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ '--exclude * --include images/*'

which also break command.

If I quote (or double quote) only `*` is not replaced and executed command look like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude '*' --include images/*

but then in debug mode I can see:

020-01-02 00:01:27,529 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/'*'

so it use wrong value with filter (not remove quotes) and results with not match.

Is there any way to escape `*` that it will not be replaced?


4 answers

1 accepted

0 votes
Answer accepted

Thanks for reply.

I tried your suggestion, but `atlassian/aws-s3-deploy` doesn't support `Advanced techniques for writing pipes

Finally, I resolve problem using bash equals-separated options notation writing `--exclude=*` instead of `--exclude *` (`=` instead of space) which doesn't extend glob and results command like:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude=* --include=images/*

Ah nice, I was not aware that aws cli supports this mode but actually checked it in their docs which did not outline it at the command, I think they cover it in the general description, yeah found it:

Optionally, you can optionally separate the parameter name from the value with an equals sign (=) instead of a space. This is typically necessary only if the value of the parameter starts with a hyphen.

From: Specifying Parameter Values for the AWS CLI

Another last resort might be to provide a json template as many aws cli commands support those, but would be more work. Just if someone stumbles of it here. It is in the same link: Using JSON for Parameters .

0 votes
Pramodh Community Leader Jan 01, 2020

Try \*

I tried and asterisk is not replaced but invlaid value is passed to aws command:

aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude \* --include images/*

and filtering faild with results:

2020-01-02 06:34:16,533 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/\*

Looks like the problem is with glob expansions with the EXTRA_ARGS parameter of the pipe.

Try with a different notation in the pipelines YAML:

  EXTRA_ARGS: ['--exclude', '*', ...]

This is in reference to: Advanced techniques for writing pipes 

If the Pipe you use follows this reference (I would guess so as it looks like you're using the one offered by Atlassian here) , this might be more explicit and better to control. But you need to try as I don't know for sure.

Additionally I remember a similar issue not with glob expansion but with variable substitution and EXTRA_ARGS: Way to use "---include-from" in rsync deploy - In the end, the variable substitution only worked by not putting the variable name into curly brackets:

EXTRA_ARGS: '--include-from=$BITBUCKET_CLONE_DIR/deploy-include.txt --exclude=*'

Note that this is the rsync utility not the aws utility, but also a pipe from Atlassian.

So why not give the different notation a try and post the outcome. This might shed some more light.

This should also help to find out if there is a general problem with the pipe regarding --exclude / --include patterns (vendor docs about the patterns) which should then be reported against the concrete pipe repository (atlassian/aws-s3-deploy).

Suggest an answer

Log in or Sign up to answer
Community showcase
Published in Bitbucket Pipelines

Bitbucket Pipelines Runners is now in open beta

We are excited to announce the open beta program for self-hosted runners. Bitbucket Pipelines Runners is available to everyone. Please try it and let us know your feedback. If you have any issue...

3,939 views 68 21
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you