Pipeline takes forever on Build setup step

[Piotr Bakun]

Recently we've noticed significant time increase in our pipelines. 

It happens randomly, but the "Build setup" step can take even 20 mins, It seems to be stuck on `Downloading cache` step.

We run our pipelines on self-hosted docker runners. I recently updated them to v3.6.0. but it didn't help.

We've verified that the machine they run on have stable internet connection.

What might be the reason for Build step to run so long?

[Patrik S]

Hello @Piotr Bakun ,

and welcome to the Community!

Starting on version 3.0, the runner implements a more reliable and faster management of artifacts and caches, which directly interacts with AWS S3.

Since the cache and artifacts are stored in S3 the issue might be slow downloads on the Runner where there is a connection problem between the runner's host and S3. 

In this case, would it be possible for you run some tests on your runner's host to try and download some objects from S3 (us-east-1) to test the speed?

This would help to narrow down the issue and confirm if it's network related.

Also, a new version of runners (3.8.0) was recently released with some tweaks in network retries/timeouts. I would recommend trying to update to the newest version and check if there's any improvement.

Thank you, @Piotr Bakun !

Patrik S

[Piotr Bakun]

Hi @Patrik S ,

Sorry for keeping you waiting so long.

I performed several download tests from an S3 bucket located in us-east-1. The results were somewhat random. I tested it with a ~200MB file.

Typically, in the morning, the download took a very long time using both aws-cli and a presigned URL. However, as I downloaded the file multiple times, the waiting time decreased over subsequent attempts.

The differences were quite significant — it could take up to 10 minutes or as little as 10 seconds. I saved logs from these downloads and can send them to you via email (unfortunately, it seems attachments aren't supported here). This behavior also aligns with my observations from the runner, where the cache download step can take anywhere from a few seconds to 20 minutes.

Please also note that my runner's host is located in eu-west.

I have also updated runners to 3.8.0, but unfortunately it did not help.

[Patrik S]

Hello @Piotr Bakun ,

No worries about the delay! Thank you for sharing the results of your testing.

As you were able to reproduce the same issue outside the runners, this suggests a connection issue inside AWS that is causing this slowness/intermittent behavior.

Since this is not isolated to the runner itself but general to your host, I would suggest contacting AWS to help with the investigation of why your eu-west instance is taking that long to download files from us-east. Once that is sorted out, you'll likely see improvements in the runners' download speed as well.

Thank you, @Piotr Bakun !

Patrik S

[Piotr Bakun]

Hi @Patrik S

You might have understood that we have our runners on AWS, when I mentioned their located in eu-west. This is not particularly true.

We host our runners on our server and don't use AWS in any way. I don't see why AWS would want to solve download speed problem, if we are not their customers.

I will run some more tests on different connections, but all of them will be from western europe.

[Patrik S]

Hello @Piotr Bakun ,

You're correct! When you mentioned eu-west, I mistakenly assumed your runners were deployed to AWS, which is why I suggested reaching out to AWS.

As you mentioned, those are hosted on your own server instead. Since the slowness issue was confirmed not to be isolated to runners but also affects direct connections from your host to AWS S3, I would recommend reaching out to your hosting platform or ISP that provides connectivity from your server to the internet so they can help investigate the reason for the slowness you're experiencing on that server.

Thank you, @Piotr Bakun !
Patrik S