I'm trying to use `atlassian/aws-s3-deploy` with `EXTRA_ARGS` pipe variable like: `--exclude * --include images/*` but single `*` character in `--exclude` argument is replaced with all files listed in the current category separated by a space. Final command run by pipe should look like: aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude * --include images/* But in debug mode I see: aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude app.php css images js --include 'images/*' As you can see, `*` character was replaced with file list which break command to work. If I quote (or double quote) `EXTRA_ARGS` the `*` is not replaced but executed command look like: aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ '--exclude * --include images/*' which also break command. If I quote (or double quote) only `*` is not replaced and executed command look like: aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude '*' --include images/* but then in debug mode I can see: 020-01-02 00:01:27,529 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/'*' so it use wrong value with filter (not remove quotes) and results with not match. Is there any way to escape `*` that it will not be replaced?

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How to escape `*` character in pipe variables?

Łukasz Świerżewski

I'm trying to use `atlassian/aws-s3-deploy` with `EXTRA_ARGS` pipe variable like: `--exclude * --include images/*` but single `*` character in `--exclude` argument is replaced with all files listed in the current category separated by a space.

Final command run by pipe should look like:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude * --include images/*</span>

But in debug mode I see:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude app.php css images js --include 'images/*'</span>

As you can see, `*` character was replaced with file list which break command to work.

If I quote (or double quote) `EXTRA_ARGS` the `*` is not replaced but executed command look like:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ '--exclude * --include images/*'</span>

which also break command.

If I quote (or double quote) only `*` is not replaced and executed command look like:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude '*' --include images/*</span>

but then in debug mode I can see:

<span>020-01-02 00:01:27,529 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/'*'</span>

so it use wrong value with filter (not remove quotes) and results with not match.

Is there any way to escape `*` that it will not be replaced?

Accepted answers

Łukasz Świerżewski

Thanks for reply.

I tried your suggestion, but `atlassian/aws-s3-deploy` doesn't support `Advanced techniques for writing pipes

Finally, I resolve problem using bash equals-separated options notation writing `--exclude=*` instead of `--exclude *` (`=` instead of space) which doesn't extend glob and results command like:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude=* --include=images/*</span>

All comments

Pramodh M

Try \*

Łukasz Świerżewski

I tried and asterisk is not replaced but invlaid value is passed to aws command:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude \* --include images/*</span>

and filtering faild with results:

<span>2020-01-02 06:34:16,533 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/\*</span>

ktomk

Looks like the problem is with glob expansions with the EXTRA_ARGS parameter of the pipe.

Try with a different notation in the pipelines YAML:

variables:
  EXTRA_ARGS: ['--exclude', '*', ...]

This is in reference to: Advanced techniques for writing pipes

If the Pipe you use follows this reference (I would guess so as it looks like you're using the one offered by Atlassian here) , this might be more explicit and better to control. But you need to try as I don't know for sure.

Additionally I remember a similar issue not with glob expansion but with variable substitution and EXTRA_ARGS: Way to use "---include-from" in rsync deploy - In the end, the variable substitution only worked by not putting the variable name into curly brackets:

EXTRA_ARGS: '--include-from=$BITBUCKET_CLONE_DIR/deploy-include.txt --exclude=*'

Note that this is the rsync utility not the aws utility, but also a pipe from Atlassian.

So why not give the different notation a try and post the outcome. This might shed some more light.

This should also help to find out if there is a general problem with the pipe regarding --exclude / --include patterns (vendor docs about the patterns) which should then be reported against the concrete pipe repository (atlassian/aws-s3-deploy).

Łukasz Świerżewski

Thanks for reply.

I tried your suggestion, but `atlassian/aws-s3-deploy` doesn't support `Advanced techniques for writing pipes

Finally, I resolve problem using bash equals-separated options notation writing `--exclude=*` instead of `--exclude *` (`=` instead of space) which doesn't extend glob and results command like:

<span>aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude=* --include=images/*</span>

ktomk

Ah nice, I was not aware that aws cli supports this mode but actually checked it in their docs which did not outline it at the command, I think they cover it in the general description, yeah found it:

Optionally, you can optionally separate the parameter name from the value with an equals sign (=) instead of a space. This is typically necessary only if the value of the parameter starts with a hyphen.

From: Specifying Parameter Values for the AWS CLI

Another last resort might be to provide a json template as many aws cli commands support those, but would be more work. Just if someone stumbles of it here. It is in the same link: Using JSON for Parameters .

Szabolcs Fodor

Great! Thanks!
It seems the same trick seems to work with the Azure CLI pipe (atlassian/azure-cli-run:1.1.0) as well. E.g.:

az cdn endpoint purge --resource-group=rg --profile-name=profile --name=endpoint --content-paths=/*

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket-pipelines

variable

aws-code