50% time increase in 1x steps running tests in parallel

[Martin Defeche]

Hello,

Since the rollout of the cloud runtime 2, we noticed a 50% time increase when running python tests in parallel on a 1x step (~15min to ~23min).

We identified the runtime based on the log at the end of the first step:

Cloud runtime: 1 # before
Cloud runtime: 2 # now

It was clearly stated that there won't be a decrease in compute for 1x steps

Important: We want to emphasise that this is NOT a reduction in compute for 1x steps. In fact, 2x steps will see a significant INCREASE in compute allocation. The move to a dedicated CPU model means each visible CPU will provide more actual compute power.

Is there something to tweak to get the same results as before ?

We were already impacted by the decision to restrict the `atlassian-ip-ranges` to 4x/8x steps which forced us to move certain steps to more expensive sizes.

With those two problems, our build minutes usage has significantly increased, and our bill too.

Are there plans to mitigate the price increase for customers caused by this migration ?

Thank you

[Edmund Munday]

If anyone else is seeing measurable performance degradations of this kind of scale, please reach out to the Bitbucket Support team so that the engineering team can troubleshoot the underlying cause.

We do expect some customer to see a change in build speeds, but not an overall degradation.

The reason for this is we have moved to a much more encapsulated model when it comes to assigning CPU resources to each build, meaning builds should be more consistent now than in the past.

This means that some of your builds in the past may have been marginally faster, but an equivalent amount would have been marginally slower, leading to an average build time that is effectively equal.

For context - we conducted extensive performance testing and tuning of the new runtime over the past 3 months covers several million builds to ensure that performance was tuned to provide consistent performance.

[Swetha Boyapati]

HI @Martin Defeche, 

Good day!!I'm Swetha from Bitbucket Cloud Support. 

We have created a support ticket for this issue as we need to look into the builds which would need us accessing the repo. You'll get an email notification with the support ticket details. 

Cheers, 
Swetha Boyapati

[Kirsty Mullen]

We're experiencing the same across our pipelines. It's very noticable in our vitest step (~1-2min to ~5min). Our PR pipeline has doubled in its average time overall (~6 to ~12).

Local performance is unchanged and we cannot attribute it to any code change. The cloud runtime log does indeed change on the first affected pipeline to 2.

I'll create a support ticket as advised but wanted to echo what has been raised as it's not an isolated occurrence.

Kind regards,