old azcopy in Azure Storage Deploy v0.5.0

[Grzegorz Meger]

hello, i cant use pipeline v0.5.0 to deploy files to container '$web' which is default name for static web container in Azure storage.

pipeline uses 7.2.0 version of azcopy, and this bug was fixed in 7.3.0 version of azcopy

https://github.com/Azure/azure-storage-net-data-movement/issues/133

please update azcopy to new version 

thanks

Greg

[Steven Vaccarella · Accepted solution]

Hi Greg,

Thanks for bringing this issue to our attention. We've been in touch with Microsoft and they've now released a new version of the pipe (1.1.1) that uses azcopy 7.3.0.

[Nico Lindemann]

Hello, the issue seems to stay. Are there any additional problem known? 

[Steven Vaccarella]

Could you post the exact yaml snippet you're using to invoke the pipe, along with any relevant error output that you get when executing it?

[Nico Lindemann]

Snippet:

 - pipe: microsoft/azure-storage-deploy:1.1.1
   variables:
     SOURCE: 'build/'
     DESTINATION: 'https://xxxxxxxxxxxx.blob.core.windows.net/$web/'
     DESTINATION_SAS_TOKEN: $AZURE_STORAGE_SAS_TOKEN
     EXTRA_ARGS: '--verbose --set-content-type'

The problem seems to be rooted in the use of "$web". I tried various ways to escape the '$' but only with an environment variable i was able to get the pipe running without any error and correct behaviour. 

[Steven Vaccarella]

I'm glad you've got it working. Quoting/escaping of special characters like '$' in pipe variables can be quite tricky because there are always at least two rounds of processing that can cause the characters to be interpreted as something other than a literal value. In the case of this pipe, the way it is currently implemented, there are actually three rounds of processing (two to pass the value into the pipe and a third round within the pipe itself).

Please have a look here for general information on quoting and escaping of pipe variables: https://community.atlassian.com/t5/Bitbucket-Pipelines-questions/How-do-I-properly-quote-Pipelines-pipe-variables-and-escape/qaq-p/1021713 This describes the first two rounds of processing.

So based on that information, it would appear that you need to do this to get the correct value into the pipe:

DESTINATION: 'https://xxxxxxxxxxxx.blob.core.windows.net/\$web/'

However as I mentioned this particular pipe actually does another round of processing which will cause the "$web" value to be treated as an environment variable expansion. So I believe the correct quoting/escaping for your example (for version 1.1.1 of the pipe) would be this:

DESTINATION: 'https://xxxxxxxxxxxx.blob.core.windows.net/\\\$web/'

I know that this is confusing and difficult to get right (I'm not even 100% certain myself that my suggestion will work without testing it). We're currently looking at how we can improve the pipe implementations to minimise the amount of escaping that's required.

[Ilija Vidović]

A potentially useful information regarding this:

I've managed to fix the issue by setting my destination environment variable to:

https://xxxxxxxxxxxx.blob.core.windows.net/\$web/

Without backslash or with three of them it did not work.

Addititionally, I'd suggest using EXTRA_ARGS variable from this pipe, and setting your environment variable for that to

--set-content-type

As that will keep original file type (otherwise you might ran into an issue that when opening your static website, it starts a download, because without that argument it sets default blob type to application/octet-stream, which you probably do not want).

However, all of this seems to be rather useless if your build is producing files with different hashes (in file name), like create-react-app / WebPack does - because thispipe  does not delete those previous files (and offers no flag to do so that I've found), you'd end up with all older files taking up your storage for no reason.

[Steven Vaccarella]

Hi Ilija,

Thanks for providing that extra information.

To clarify for others who might be confused (as I was initially), my example above is appropriate if you're configuring the destination (containing the dollar sign) directly in the yaml file. I'm guessing what Ilija did is put the destination (containing the dollar sign) into a repository variable using the Bitbucket UI which means that the first two rounds of processing (yaml and variable expansion) don't apply. Which is why only a single backslash is required to survive the final round of processing within the pipe.

[vlad]

The only option that worked for me was three slashes and no environment variables used.


It would be great if @microsoft could add this as a warning to their storage pipeline so people don't continually run into this issue.@Stephen Sifers 

[Stephen Sifers]

Hello Grzegorz,

Thank you for finding and reporting this issue with Bitbucket pipelines and azcopy version 7.2. I have submitted a feature request for this to have it reviewed. You may find the created feature request at Issue #18211.

We would suggest you vote and watch this issue so you may receive future updates on its status.

Regards,
Stephen Sifers