Uploading artifact from Bitbucket pipeline is extremely slow

[Lasse Lehtinen]

Hi,

We are running Bitbucket Pipelines on a PHP project where the dependencies are installed with composer. I noticed that uploading the artifact takes unsually long. Compression takes 9 seconds but uploading a relatively small 37Mb file takes 141 seconds. The transfer speed is 2 Mbit/s. The runner is run locally, but I could not find any network related reasons why the upload would be so slow. I noticed that latest 3.X versions are now using S3 buckets to store the artifacts.

Compressed files matching artifact pattern vendor/** to 36.7 MiB in 9 seconds
Uploading artifact of 36.7 MiB Successfully uploaded artifact in 141 seconds

Downloading the same artifact back on another task is much faster (9 Mbit/s):

"vendor/**": Downloaded 36.7 MiB in 34 seconds

On another project I found older pipelines in which the upload was way faster. Seems like uploading artifacts was changed to S3 starting from version 3.0.0 according to the changelog.

Version 2.1.0

Uploading artifact of 30.4 MiB
Successfully uploaded artifact in 22 seconds

Version 3.15.0

Uploading artifact of 33.2 MiB
Successfully uploaded artifact in 117 seconds 

[Theodora Boudale]

Hi Lasse and welcome to the community.

I haven't been able to reproduce this. I tried running a few builds (on a self-hosted runner) with an artifact of about 30 MB when compressed and the upload was completed in less than 15 seconds.

Artifacts are uploaded to S3 in us-east-1 and us-west-2 regions. One thing that comes to mind is high latency if the server where the runner is running is far away from the US. Is that the case for your server?

Kind regards,
Theodora

[Lasse Lehtinen]

This is correct, we are located in Finland.

Any way to manually test the upload speed? I do have an AWS account so technically I could create test buckets in different regions to see the difference.

Another question is that would it be possible to skip the uploading the artifacts to an external S3 bucket and use local storage instead?

[Theodora Boudale]

Hi Lasse,

If you have the option of creating a server located in the US, then I suggest doing that and observing if the upload completes faster. If you still experience issues, you can create a ticket with the support team for further investigation. You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.

I'm afraid that that there is no option to use local storage for artifacts at the moment. We have a feature request for what you're asking:

• https://jira.atlassian.com/browse/BCLOUD-21396

You can add your vote to that feature request (by selecting the link Vote for this issue) to increase the chances of this being implemented. You are more than welcome to leave feedback, and you can also add yourself as a watcher (by selecting the link Start watching this issue) if you'd like to be notified via email on updates.

Implementation of features is done as per our policy here and any updates will be posted in the feature request.

Kind regards,
Theodora

[Lasse Lehtinen]

Thanks, I voted for that feature request. Because of firewall issues, we have run the pipeline locally. So we don't really have an easy option to host the runner anywhere else.

[Theodora Boudale]

You are welcome, Lasse.

I discussed this further with the development team and they confirmed that runners can experience high latency when running cross-region with Pipelines (which is based in the US).

I'm afraid that there is no workaround at the moment, other then hosting the runners in the US region.

We have a request for supporting S3 transfer acceleration for faster uploads and downloads with self-hosted runners:

• https://jira.atlassian.com/browse/BCLOUD-23542

You can add your vote to this request as well to express your interest.

Kind regards,
Theodora

[Lasse Lehtinen]

Hi,

Thanks, upvoted. I tested from our co-location using the AWS provided tester. Seems like there would be some minor improvements but nothing too significant, around 3-10 percent increase to US-EAST-1 and US-WEST-2.

https://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html

[Tiago Jesus]

Indeed, after version 3.0.0 the artifacts uploading is very slow and sometimes fails.

[Lasse Lehtinen]

I also had the timeout issues first. Increasing the timeout helped so that atleast we can run the pipelines, but just way slower than with the older versions of the pipeline runner.