Speed up sharing of artifacts

Cameron Ferguson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 5, 2012

We have noticed in our bamboo setup that larger artifacts take a VERY long time to share. All of our agents and the actual bamboo server are on a gigabit network and are very high end computers (dual hex core xeons hyperthreaded, 24 gigs of ram, etc)

Publishing an artifact of 250mb doesn't take any time at all.. but when an agent needs that artifact, the "preparing artifact for use" section takes roughly 3-4 minutes for the artifact to become available.

I've manually tested a secure copy from "agent to agent", "agent to server" and "server to agent", and it only takes about 3-5 seconds for the data to be transfered... Is there some option that can cause this? I am not sure why this takes so long for bamboo to "prepare the artifact for use"

I'd appreciate it!

Thanks!

1 answer

1 accepted

0 votes
Answer accepted
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 5, 2012

Hmm, are you using standalone distribution (Jetty) or Tomcat? Also, what OS is that? There are known transfer performance problems with Win2008 on Tomcat.

With that much of a difference between publish and fetch, your question qualifies for support.atlassian.com, because it looks like a bug.

Cameron Ferguson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 6, 2012

I just found it: jetty/7.2.1.v20101111

Cameron Ferguson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 6, 2012

We use a few different OS types...

Windows XP 64 bit

CentOS 5.5 64 bit

CentOS 5.3 64 bit

RedHat 6.0 64 bit

Each of these OS's have the same slowdown... though it seems that WindowsXP takes maybe an extra minute over the Linux distros we use.

I am not sure what standalone distribution we are using... where would I find that?

Cameron Ferguson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 6, 2012

Let me also add something that may be the reason this is happening now that I think about it.

We compile about 11 different versions of a few of our products (different compilers, different OS, different architectures, etc). We had an initial design issue with bamboo and our perforce server where when we had all 11 of these different build jobs pull, we would have errors with perforce that so much was being pulled at once by the same user from the same location. To solve this issue, we basically designed our first stage to pull our repository from perforce, and then share that as an artifact (The copy pattern was just ** ). We do not zip this artifact up or tar it or anything, we just share the repository itself and all files.

In the 2nd stage, our build stage, we would depend upon this artifact. This way for an entire build, we pull the repository once, and then share it with other jobs.

For one of our repositories that takes quite a long time, it's roughly 3600 files that total up around 250mb. Could the reason for the slow up be because there are so many files that are shared and not necessarily the size?

Cameron Ferguson
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 7, 2012

Zipping up the files as a single file to be shared seems to have sped up the process... our build times prior to this change were about 10 minutes... they are now down to 5 with the zip feature. Seems bamboo really doesn't like sharing thousands of files as a single artifact.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events