Speed up sharing of artifacts

We have noticed in our bamboo setup that larger artifacts take a VERY long time to share. All of our agents and the actual bamboo server are on a gigabit network and are very high end computers (dual hex core xeons hyperthreaded, 24 gigs of ram, etc)

Publishing an artifact of 250mb doesn't take any time at all.. but when an agent needs that artifact, the "preparing artifact for use" section takes roughly 3-4 minutes for the artifact to become available.

I've manually tested a secure copy from "agent to agent", "agent to server" and "server to agent", and it only takes about 3-5 seconds for the data to be transfered... Is there some option that can cause this? I am not sure why this takes so long for bamboo to "prepare the artifact for use"

I'd appreciate it!

Thanks!

1 answer

1 accepted

0 votes
Accepted answer

Hmm, are you using standalone distribution (Jetty) or Tomcat? Also, what OS is that? There are known transfer performance problems with Win2008 on Tomcat.

With that much of a difference between publish and fetch, your question qualifies for support.atlassian.com, because it looks like a bug.

I just found it: jetty/7.2.1.v20101111

We use a few different OS types...

Windows XP 64 bit

CentOS 5.5 64 bit

CentOS 5.3 64 bit

RedHat 6.0 64 bit

Each of these OS's have the same slowdown... though it seems that WindowsXP takes maybe an extra minute over the Linux distros we use.

I am not sure what standalone distribution we are using... where would I find that?

Let me also add something that may be the reason this is happening now that I think about it.

We compile about 11 different versions of a few of our products (different compilers, different OS, different architectures, etc). We had an initial design issue with bamboo and our perforce server where when we had all 11 of these different build jobs pull, we would have errors with perforce that so much was being pulled at once by the same user from the same location. To solve this issue, we basically designed our first stage to pull our repository from perforce, and then share that as an artifact (The copy pattern was just ** ). We do not zip this artifact up or tar it or anything, we just share the repository itself and all files.

In the 2nd stage, our build stage, we would depend upon this artifact. This way for an entire build, we pull the repository once, and then share it with other jobs.

For one of our repositories that takes quite a long time, it's roughly 3600 files that total up around 250mb. Could the reason for the slow up be because there are so many files that are shared and not necessarily the size?

Zipping up the files as a single file to be shared seems to have sped up the process... our build times prior to this change were about 10 minutes... they are now down to 5 with the zip feature. Seems bamboo really doesn't like sharing thousands of files as a single artifact.

Suggest an answer

Log in or Sign up to answer
Community showcase
Asked Thursday in Jira Ops

I'm John Allspaw, Ask Me Anything about incident analysis and postmortems

I'm John Allspaw, co-founder of   Adaptive Capacity Labs, where we help teams use their incidents to learn and improve. We bring research-driven methods and approaches to drive effective inciden...

3,489 views 8 7
View question

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you