Use artifact in second stage without publish

Question

Hello,

I'm trying to set up a plan with stages which runs integration tests. I've set-up my structure as described in the Bamboo Best Practice - Using stages article.

* Stage 1: build & publish a shared artifact* Default Job

* Stage 2: dependent on artifact, run test* Test Job 1
* Test Job 2

I have multiple, separated integration tests projects which are allowed to run in parallel and if one fails, the other ones should be allowed to continue or start. This means that combining everything in 1 stage is not really an option, because if the build fails, none of them should run, but if one fails, the other ones should be allowed to run.

Since I'm using Bamboo Cloud, I have little to no control over where the artifacts show up or how fast the upload happens. And it's slow. Very slow. Unacceptable slow.

My artifact also contains some sample files. They are part of my project setup, required to run the integration tests and thus embedded in the artifact. The total size of my artifact is thus around 150 MB, which I can get down to 80 MB when I create a ZIP file instead of uploading the individual items.

But that still takes about 10-15 minutes to publish. And that doesn't work for me. This means the total time of a single test build takes around 20 minutes, where the actual build and the test run only take about 2 minutes. And that's not acceptable if you want to run them as part of a continuous build plan.

I came across BAM-15572 which mentions a possible work around by moving the instances to the US West region, but it still isn't ideal.

So I was wondering:

* Is there a better strategy?
* Can I improve the upload speed?
* Can I somehow use the build result from stage 1 in the following stages without the need to publish the artifact?

Jensen Somers · Answer

Since I've moved the AMI to the US West region, my upload times have decreased. It only took 10 second to upload a 60 MB artifact, which at the moment is acceptable.

But I will take your advice into consideration. At some point it would be advisable to keep artifacts around longer than 30 days (which I believe is the default Bamboo Cloud retention policy), so uploading it into my own storage container would be more advisable.

Alex Soto_a4l4ig · Answer

Instead of using the zip file as the artifact, instead push the artifact yourself to S3 (or some other fast/shared storage you have access to) via some scripting. Then write the URL to that artifact in a simple text file.  Use that text file as your artifact.  Then it's published quickly and you read the URL from the artifact in a script in further stages to pull the zip.

I have no direct experience with large artifacts myself, but I do publish a simple file as an artifact and I've never experienced slowness with it.

Good luck