nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Uploading artifact from Bitbucket pipeline is extremely slow

Lasse Lehtinen

Hi,

We are running Bitbucket Pipelines on a PHP project where the dependencies are installed with composer. I noticed that uploading the artifact takes unsually long. Compression takes 9 seconds but uploading a relatively small 37Mb file takes 141 seconds. The transfer speed is 2 Mbit/s. The runner is run locally, but I could not find any network related reasons why the upload would be so slow. I noticed that latest 3.X versions are now using S3 buckets to store the artifacts.

Compressed files matching artifact pattern vendor/** to 36.7 MiB in 9 seconds
Uploading artifact of 36.7 MiB Successfully uploaded artifact in 141 seconds

Downloading the same artifact back on another task is much faster (9 Mbit/s):

"vendor/**": Downloaded 36.7 MiB in 34 seconds

On another project I found older pipelines in which the upload was way faster. Seems like uploading artifacts was changed to S3 starting from version 3.0.0 according to the changelog.

Version 2.1.0

<span>Uploading artifact of 30.4 MiB<br>Successfully uploaded artifact in 22 seconds</span>

Version 3.15.0

 <span>Uploading artifact of 33.2 MiB<br>Successfully uploaded artifact in 117 seconds</span>

Accepted answers

All comments

Tiago Jesus_mbgcfz

Indeed, after version 3.0.0 the artifacts uploading is very slow and sometimes fails.

Lasse Lehtinen

I also had the timeout issues first. Increasing the timeout helped so that atleast we can run the pipelines, but just way slower than with the older versions of the pipeline runner.

Theodora Boudale

Hi Lasse and welcome to the community.

I haven't been able to reproduce this. I tried running a few builds (on a self-hosted runner) with an artifact of about 30 MB when compressed and the upload was completed in less than 15 seconds.

Artifacts are uploaded to S3 in us-east-1 and us-west-2 regions. One thing that comes to mind is high latency if the server where the runner is running is far away from the US. Is that the case for your server?

Kind regards,
Theodora

Lasse Lehtinen

This is correct, we are located in Finland.

Any way to manually test the upload speed? I do have an AWS account so technically I could create test buckets in different regions to see the difference.

Another question is that would it be possible to skip the uploading the artifacts to an external S3 bucket and use local storage instead?

Theodora Boudale

Hi Lasse,

If you have the option of creating a server located in the US, then I suggest doing that and observing if the upload completes faster. If you still experience issues, you can create a ticket with the support team for further investigation. You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.

I'm afraid that that there is no option to use local storage for artifacts at the moment. We have a feature request for what you're asking:

https://jira.atlassian.com/browse/BCLOUD-21396

You can add your vote to that feature request (by selecting the link Vote for this issue) to increase the chances of this being implemented. You are more than welcome to leave feedback, and you can also add yourself as a watcher (by selecting the link Start watching this issue) if you'd like to be notified via email on updates.

Implementation of features is done as per our policy here and any updates will be posted in the feature request.

Kind regards,
Theodora

Lasse Lehtinen

Thanks, I voted for that feature request. Because of firewall issues, we have run the pipeline locally. So we don't really have an easy option to host the runner anywhere else.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bitbucket

bitbucket-cloud

pipelines