Speed up sharing of artifacts

We have noticed in our bamboo setup that larger artifacts take a VERY long time to share. All of our agents and the actual bamboo server are on a gigabit network and are very high end computers (dual hex core xeons hyperthreaded, 24 gigs of ram, etc)

Publishing an artifact of 250mb doesn't take any time at all.. but when an agent needs that artifact, the "preparing artifact for use" section takes roughly 3-4 minutes for the artifact to become available.

I've manually tested a secure copy from "agent to agent", "agent to server" and "server to agent", and it only takes about 3-5 seconds for the data to be transfered... Is there some option that can cause this? I am not sure why this takes so long for bamboo to "prepare the artifact for use"

I'd appreciate it!

Thanks!

1 answer

1 accepted

0 vote

Hmm, are you using standalone distribution (Jetty) or Tomcat? Also, what OS is that? There are known transfer performance problems with Win2008 on Tomcat.

With that much of a difference between publish and fetch, your question qualifies for support.atlassian.com, because it looks like a bug.

I just found it: jetty/7.2.1.v20101111

We use a few different OS types...

Windows XP 64 bit

CentOS 5.5 64 bit

CentOS 5.3 64 bit

RedHat 6.0 64 bit

Each of these OS's have the same slowdown... though it seems that WindowsXP takes maybe an extra minute over the Linux distros we use.

I am not sure what standalone distribution we are using... where would I find that?

Let me also add something that may be the reason this is happening now that I think about it.

We compile about 11 different versions of a few of our products (different compilers, different OS, different architectures, etc). We had an initial design issue with bamboo and our perforce server where when we had all 11 of these different build jobs pull, we would have errors with perforce that so much was being pulled at once by the same user from the same location. To solve this issue, we basically designed our first stage to pull our repository from perforce, and then share that as an artifact (The copy pattern was just ** ). We do not zip this artifact up or tar it or anything, we just share the repository itself and all files.

In the 2nd stage, our build stage, we would depend upon this artifact. This way for an entire build, we pull the repository once, and then share it with other jobs.

For one of our repositories that takes quite a long time, it's roughly 3600 files that total up around 250mb. Could the reason for the slow up be because there are so many files that are shared and not necessarily the size?

Zipping up the files as a single file to be shared seems to have sped up the process... our build times prior to this change were about 10 minutes... they are now down to 5 with the zip feature. Seems bamboo really doesn't like sharing thousands of files as a single artifact.

Suggest an answer

Log in or Sign up to answer
Atlassian Community Anniversary

Happy Anniversary, Atlassian Community!

This community is celebrating its one-year anniversary and Atlassian co-founder Mike Cannon-Brookes has all the feels.

Read more
Community showcase
Renan Battaglin
Published May 18, 2017 in Bamboo

FAQ: How to Upgrade Bamboo Server

Bamboo 5.9 will no longer be supported after June 12, 2017. What does this mean? As part of our End of Life policy, Atlassian supports major versions for two years after the first major iteratio...

1,321 views 0 5
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you