Why pipelines running time has increased for the same commit?

[Daniel Simard]

I have this pipeline that ran on December 7th, 2021 for ~20 minutes:

We noticed a increase in build times so I ran the build on the same commit on December 15th, 2021 and it took ~45 minutes:

Then I ran the build on the same commit yesterday and it took ~60 minutes:

Did you notice the same thing on your builds? What could cause that?

[Craig Castle-Mead]

Hi @Daniel Simard 

https://jira.atlassian.com/browse/BCLOUD-15694 states a few possible reasons:

The two main contributors to build time variations on Pipelines are:

• External resources, like Docker images or dependencies that need to be downloaded from the internet. These sometimes vary a lot in how long they take to retrieve from external services.
• Variation in CPU availability in Pipelines. We run your builds as Docker containers in a cluster shared with other Bitbucket customers, so there can be CPU contention on our hosts during busy periods. Sometimes you get on a quiet node and your build executes quickly, other times it may have contention and complete more slowly.

On the latter, we're experimenting on ways to improve this while still allowing customers to get burst CPU usage when they need it. We're tracking this as BCLOUD-13079, so please watch that issue if you want to stay updated.

If your build process relies on third party services (package installers, rest APIs etc) then you also need to consider whether those dependent services are responding slower. I don't use Pipelines (not on Bitbucket cloud), but can you get more detailed logging on steps 2 and 3 - ideally with timestamps and see if there's a particular command that has increased and use that to continue your investigation?

CCM

[Daniel Simard]

External resources in both builds took about 2 minutes, it wasn't that.

Variation in CPU availability in Pipelines. Build time was constantly higher for a couple of weeks. It's not like there was moments during the day that the builds were as fast as before.

How it was solved. Right now, it looks like we're back to our previous low build times of a month ago. I wrote to the billing department, maybe it was that. I'll reply here if we get back to high build times.