For loop script syntax error

[Andy Furniss]

I'm trying to use the AWS CLI to invalidate my CloudFront distribution. This consists of two commands, one to create the invalidation and then one to wait for it to complete.

To do this, I need to capture the invalidation ID that AWS sends back to me and pass it to the second command. After lots of Googling I have settled on the below script. Whilst not pretty, it serves it's purpose.

pipelines:
    branches:
        Stage/development:
            - step:
                name: Invalidate
                image: cgswong/aws:aws
                script:
                  - for /f usebackq %F in (`aws cloudfront create-invalidation --distribution-id "##############" --paths /* --query "Invalidation.Id"`) do aws cloudfront wait invalidation-completed --distribution-id "##############" --id %F

The command I've written works perfectly on my machine (Windows 10) but when it's run as part of the Pipeline, I get the following error:

bash: /opt/atlassian/pipelines/agent/tmp/bashScript6992979577238911185.sh: line 5: syntax error near unexpected token `usebackq'

I've tried putting usebackq in quotes but that didn't help - same error.

Could someone help with this please or offer another solution?

[Daniel Santos · Accepted solution]

Hi @Andy Furniss

You will need to convert your command from CMD (Windows) to bash (Linux). The format of a loop in bash would be something like:

for F in $(command1); do command2 $F ;done

The command1 ideally should return values with quotes separated by space or line breaks. The quotes are only needed if the names have spaces in them because space is the separator of elements here.

The command2 would just embed the variable F on any part of it using $F to return the variable value.

I hope that helps.

[Andy Furniss]

Thanks a lot @Daniel Santos !

Sorry for the late reply, I haven't had a chance to get back on this task yet but I will let you know if I manage to get it working with your suggestion!

I figured it might be because it needed to be done with bash but I have zero bash experience so I couldn't figure out how to do it! Hopefully your method works :)

[Daniel Santos]

No problem, take your time.
Let me know if you face any issues to make that work.

[Andy Furniss]

Hi Daniel. I think I've made some progress but I've got stuck on how to pass the variable to the --id option of the AWS CLI invalidation-completed command. I believe that this must be an issue with the syntax of my command which you can see below:

for F in $(aws cloudfront create-invalidation --distribution-id ############## --paths /* --query "Invalidation.Id"); do aws cloudfront wait invalidation-completed --distribution-id ############## --id $F ;done

 The above outputs "Waiter InvalidationCompleted failed: The specified invalidation does not exist." which is obviously a AWS error.

The strange thing is, if I replace the second command with an echo, it outputs a valid invalidation ID, like so:

for F in $(aws cloudfront create-invalidation --distribution-id ############## --paths /* --query "Invalidation.Id"); do echo $F ;done

 This outputs a string wrapped in double quotes, like this: "C3JTO0AJ91B4Q0".

Is there an issue with how I'm using the $F variable in my command? Perhaps something I'm not aware of with Bash?

Also, I've noticed that if I manually run the second command with the value given when just echo'ing the output, it works. So it's just something wrong with how the value is being passed to the second command it seems.

[Andy Furniss]

Sorted it! I had to add:

--output text 

to the create-invalidation command so that it's return value didn't have the double quotes. Passing this to the second command was then successful.

I wonder if there's a way to handle the value when it's wrapped in double quotes though? Just for future reference :)

[Daniel Santos]

This is great!

I was not able to work on this yesterday, but I'm really glad that you were able to work this out.

If you think this answer can help others, please mark it as accepted.

Thank you for sharing your findings here =]

Have a good one!

[Tanishka]

you can try this post about loop in bash