Problem with Artifact Download from Windows to Linux

[David R. Cattley]

I have a plan/job that runs on a WIndows OS based agent and produces a shared artifact comprise of multiple files.

On another plan/job that runs on a Linux based agent I use the Artifact Download task to copy these artifacts into the build.

The problem is that instead of the directory tree being recreated the Artifact Download task creates "files" in the root of the build folder that have literal backslash embedded.

It would seem that the notion of "path separator" is not conveyed properly via this mechanism.

Example:

The produced (on Windows) shared artifact might be:   foo/bar/data.txt

The staged (on Linux) artifact ends up being a file named foo\bar\data.txt

This, of course, is complete nonsense.

Is there some option / trick that needs to be enable to make this work?

TIA

Dave Cattley

[David R. Cattley]

1.  Wait for Atlassian to fix it.

or

2. Use some other tool / container (e.g. ZIP, TAR, whatever) to pack the artifact files into a single object and unpack it on the destination.

[Daniel Santos]

Hi David,

There was a bug in the past (reported for our cloud instances - In that time we had Bamboo cloud) which has the exact behavior you are describing now. This is the one: Artifact folders produced on Windows are being published as files.

I was trying to reproduce the same issue in a recent Bamboo version (6.7.2) and I was unable to reproduce it.

• Could you share with us what is the Bamboo version affected?
• Could you also share how the artifact definition is declared?

[David R. Cattley]

Version: 6.6.1

The artifact is defined with a pattern (like the following):

Location:  <blank>

Pattern: **/foo/bar/*.*

[David R. Cattley]

[aside]  I lament the demise of Bamboo Cloud ...

[Daniel Santos]

[aside]  I lament the demise of Bamboo Cloud ...

After some strategy analysis our company released that Bitbucket Pipelines would be a better fit for a Cloud offer and this is why Bamboo is only offered as self hosted instances.

Getting back to the artifact thing, I've tried to reproduce this issue also in Bamboo 6.6.1 using a remote agent on Windows 10 and consuming the artifact on a Linux agent with multiple artifact definitions based on your description but no luck yet.

I've opened a support request for you so we can investigate this further and get more details from your server that could not be shared in this channel. You should receive a confirmation email from our support website. 

I would kingly ask you to share here the resolution you get from our support channel so other users that come to this thread will be able to take advantage of our work together and save their time.

Thank you!

[David R. Cattley]

More Info:

The artifacts in question are stored on S3.   The issue appears to be that when a pattern such as:

  **/foo/bar/*.*

matches an artifact for example:

dir1/dir2/foo/bar/file.ext

on a Linux Agent the resulting object in the S3 bucket is named as one would expect:

<s3-bucket-root>/<bamboo-artifact-name-path>/dir1/dir2/foo/bar/file.ext

The problem occurs when a Windows agent produces the same thing the "match" looks like this:

dir1\dir2\foo\bar\file.ext

and the object in S3 is named

<s3-bucket-root>/<bamboo-artifact-name-path>%5cdir1%5dir2%5cfoo%5cbar%5cfile.ext

where the "\" characters returned from path translation of the pattern match in Windows are now treated as literal characters in a filename instead of path separators.

My best guess is that the S3 artifact handler should be performing some sort of path normalization to the "path separator" expected by S3 (which evidently is not accepting "\").

When a Linux Agent downloads those artifacts it really has little choice to but to serialize them locally with the filename they have.  Someone could have actually *meant* to produce a filename with a backslash in it in the Linux world (though they should probably be exiled to a far away planet for doing so reserved for clever folks that also put backspace in their filenames as a "security measure" ...).

[David R. Cattley]

AWS/S3 has this to say about putting a backslash in an object key name:

Characters to Avoid

Avoid the following characters in a key name because of significant special handling for consistency across all applications.

• Backslash ("\")

• Left curly brace ("{")

• Non-printable ASCII characters (128–255 decimal characters)

• Caret ("^")

• Right curly brace ("}")

• Percent character ("%")

• Grave accent / back tick ("`")

• Right square bracket ("]")

• Quotation marks

• 'Greater Than' symbol (">")

• Left square bracket ("[")

• Tilde ("~")

• 'Less Than' symbol ("<")

• 'Pound' character ("#")

• Vertical bar / pipe ("|")

[Daniel Santos]

Thank you for sharing all those details here they are very insightful.
I'm following your support ticket with the Bamboo team and as soon we get a conclusion there we will be able to share the cause of this issue here. I can see it is related to how Bamboo is handling the Artifact storage to S3 on windows machines. We just need to check in what conditions and open a bug for this.

[David R. Cattley]

Thanks for your help getting this attention.

[Daniel Santos]

By the way, I agree that your workaround is the best way to move forward at the moment (an archive).

[David R. Cattley]

@Daniel Santos what ever became of this?  Anything?  Has it been fixed in some release since 6.9.1 ?

thx

[David R. Cattley]

Putting this into a script task after the Artifact Download task will "fixup" (move) the artifact files to the expected location(s) in the local file system.

#!/bin/bash
for _src in `find . -type f -regex '.*[\\]+.*'` ; do
 _dst="${_src//\\//}"
 mkdir -vp $(dirname $_dst)
 mv -v "$_src" "$_dst"
done

 I am a shell script novice so YMMV but this seems to have done what I need it to do.