It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Problem with Artifact Download from Windows to Linux

David R. Cattley Jan 04, 2019

I have a plan/job that runs on a WIndows OS based agent and produces a shared artifact comprise of multiple files.

On another plan/job that runs on a Linux based agent I use the Artifact Download task to copy these artifacts into the build.

The problem is that instead of the directory tree being recreated the Artifact Download task creates "files" in the root of the build folder that have literal backslash embedded.

It would seem that the notion of "path separator" is not conveyed properly via this mechanism.

Example:

The produced (on Windows) shared artifact might be:   foo/bar/data.txt

The staged (on Linux) artifact ends up being a file named foo\bar\data.txt

This, of course, is complete nonsense.

Is there some option / trick that needs to be enable to make this work?

TIA

Dave Cattley

 

3 answers

1 vote
Daniel Santos Atlassian Team Jan 07, 2019

Hi David,

There was a bug in the past (reported for our cloud instances - In that time we had Bamboo cloud) which has the exact behavior you are describing now. This is the one: Artifact folders produced on Windows are being published as files.

I was trying to reproduce the same issue in a recent Bamboo version (6.7.2) and I was unable to reproduce it.

  • Could you share with us what is the Bamboo version affected?
  • Could you also share how the artifact definition is declared?
David R. Cattley Jan 07, 2019

Version: 6.6.1

The artifact is defined with a pattern (like the following):

 

Location:  <blank>

Pattern: **/foo/bar/*.*

David R. Cattley Jan 07, 2019

[aside]  I lament the demise of Bamboo Cloud ...

Daniel Santos Atlassian Team Jan 08, 2019

[aside]  I lament the demise of Bamboo Cloud ...

After some strategy analysis our company released that Bitbucket Pipelines would be a better fit for a Cloud offer and this is why Bamboo is only offered as self hosted instances.

Getting back to the artifact thing, I've tried to reproduce this issue also in Bamboo 6.6.1 using a remote agent on Windows 10 and consuming the artifact on a Linux agent with multiple artifact definitions based on your description but no luck yet.

I've opened a support request for you so we can investigate this further and get more details from your server that could not be shared in this channel. You should receive a confirmation email from our support website. 

I would kingly ask you to share here the resolution you get from our support channel so other users that come to this thread will be able to take advantage of our work together and save their time.

Thank you!

David R. Cattley Jan 08, 2019

More Info:

The artifacts in question are stored on S3.   The issue appears to be that when a pattern such as:

  **/foo/bar/*.*

matches an artifact for example:

dir1/dir2/foo/bar/file.ext

on a Linux Agent the resulting object in the S3 bucket is named as one would expect:

<s3-bucket-root>/<bamboo-artifact-name-path>/dir1/dir2/foo/bar/file.ext

The problem occurs when a Windows agent produces the same thing the "match" looks like this:

dir1\dir2\foo\bar\file.ext

and the object in S3 is named

<s3-bucket-root>/<bamboo-artifact-name-path>%5cdir1%5dir2%5cfoo%5cbar%5cfile.ext

where the "\" characters returned from path translation of the pattern match in Windows are now treated as literal characters in a filename instead of path separators.

My best guess is that the S3 artifact handler should be performing some sort of path normalization to the "path separator" expected by S3 (which evidently is not accepting "\").

 

When a Linux Agent downloads those artifacts it really has little choice to but to serialize them locally with the filename they have.  Someone could have actually *meant* to produce a filename with a backslash in it in the Linux world (though they should probably be exiled to a far away planet for doing so reserved for clever folks that also put backspace in their filenames as a "security measure" ...).

David R. Cattley Jan 08, 2019

AWS/S3 has this to say about putting a backslash in an object key name:

Characters to Avoid

Avoid the following characters in a key name because of significant special handling for consistency across all applications.

  • Backslash ("\")

  • Left curly brace ("{")

  • Non-printable ASCII characters (128–255 decimal characters)

  • Caret ("^")

  • Right curly brace ("}")

  • Percent character ("%")

  • Grave accent / back tick ("`")

  • Right square bracket ("]")

  • Quotation marks

  • 'Greater Than' symbol (">")

  • Left square bracket ("[")

  • Tilde ("~")

  • 'Less Than' symbol ("<")

  • 'Pound' character ("#")

  • Vertical bar / pipe ("|")

Daniel Santos Atlassian Team Jan 09, 2019

Thank you for sharing all those details here they are very insightful.
I'm following your support ticket with the Bamboo team and as soon we get a conclusion there we will be able to share the cause of this issue here. I can see it is related to how Bamboo is handling the Artifact storage to S3 on windows machines. We just need to check in what conditions and open a bug for this.

David R. Cattley Jan 09, 2019

Thanks for your help getting this attention.

Daniel Santos Atlassian Team Jan 09, 2019

By the way, I agree that your workaround is the best way to move forward at the moment (an archive).

1 vote
David R. Cattley Jan 08, 2019

1.  Wait for Atlassian to fix it.

or

2. Use some other tool / container (e.g. ZIP, TAR, whatever) to pack the artifact files into a single object and unpack it on the destination.

0 votes
David R. Cattley Jan 14, 2019

Putting this into a script task after the Artifact Download task will "fixup" (move) the artifact files to the expected location(s) in the local file system.

#!/bin/bash
for _src in `find . -type f -regex '.*[\\]+.*'` ; do
_dst="${_src//\\//}"
mkdir -vp $(dirname $_dst)
mv -v "$_src" "$_dst"
done

 I am a shell script novice so YMMV but this seems to have done what I need it to do.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published in Next-gen

Introducing subtasks for breaking down work in next-gen projects

Teams break work down in order to help simplify complex tasks. This is often done iteratively, with tasks being broken down into smaller tasks and so on until the work is accurately captured in well-...

920 views 12 15
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you