I'm trying to use `atlassian/aws-s3-deploy` with `EXTRA_ARGS` pipe variable like: `--exclude * --include images/*` but single `*` character in `--exclude` argument is replaced with all files listed in the current category separated by a space.
Final command run by pipe should look like:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude * --include images/*
But in debug mode I see:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude app.php css images js --include 'images/*'
As you can see, `*` character was replaced with file list which break command to work.
If I quote (or double quote) `EXTRA_ARGS` the `*` is not replaced but executed command look like:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ '--exclude * --include images/*'
which also break command.
If I quote (or double quote) only `*` is not replaced and executed command look like:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude '*' --include images/*
but then in debug mode I can see:
020-01-02 00:01:27,529 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/'*'
so it use wrong value with filter (not remove quotes) and results with not match.
Is there any way to escape `*` that it will not be replaced?
Thanks for reply.
I tried your suggestion, but `atlassian/aws-s3-deploy` doesn't support `Advanced techniques for writing pipes
Finally, I resolve problem using bash equals-separated options notation writing `--exclude=*` instead of `--exclude *` (`=` instead of space) which doesn't extend glob and results command like:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude=* --include=images/*
Ah nice, I was not aware that aws cli supports this mode but actually checked it in their docs which did not outline it at the command, I think they cover it in the general description, yeah found it:
Optionally, you can optionally separate the parameter name from the value with an equals sign (=) instead of a space. This is typically necessary only if the value of the parameter starts with a hyphen.
From: Specifying Parameter Values for the AWS CLI
Another last resort might be to provide a json template as many aws cli commands support those, but would be more work. Just if someone stumbles of it here. It is in the same link: Using JSON for Parameters .
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Great! Thanks!
It seems the same trick seems to work with the Azure CLI pipe (atlassian/azure-cli-run:1.1.0) as well. E.g.:
az cdn endpoint purge --resource-group=rg --profile-name=profile --name=endpoint --content-paths=/*
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Looks like the problem is with glob expansions with the EXTRA_ARGS parameter of the pipe.
Try with a different notation in the pipelines YAML:
variables:
EXTRA_ARGS: ['--exclude', '*', ...]
This is in reference to: Advanced techniques for writing pipes
If the Pipe you use follows this reference (I would guess so as it looks like you're using the one offered by Atlassian here) , this might be more explicit and better to control. But you need to try as I don't know for sure.
Additionally I remember a similar issue not with glob expansion but with variable substitution and EXTRA_ARGS: Way to use "---include-from" in rsync deploy - In the end, the variable substitution only worked by not putting the variable name into curly brackets:
EXTRA_ARGS: '--include-from=$BITBUCKET_CLONE_DIR/deploy-include.txt --exclude=*'
Note that this is the rsync utility not the aws utility, but also a pipe from Atlassian.
So why not give the different notation a try and post the outcome. This might shed some more light.
This should also help to find out if there is a general problem with the pipe regarding --exclude / --include patterns (vendor docs about the patterns) which should then be reported against the concrete pipe repository (atlassian/aws-s3-deploy).
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I tried and asterisk is not replaced but invlaid value is passed to aws command:
aws s3 sync /opt/atlassian/pipelines/agent/build s3://bucket_name/ --exclude \* --include images/*
and filtering faild with results:
2020-01-02 06:34:16,533 - MainThread - awscli.customizations.s3.filters - DEBUG - /opt/atlassian/pipelines/agent/build/app.php did not match exclude filter: /opt/atlassian/pipelines/agent/build/\*
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Try \*
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.