50% time increase in 1x steps running tests in parallel

Question

Hello, Since the rollout of the cloud runtime 2, we noticed a 50% time increase when running python tests in parallel on a 1x step (~15min to ~23min). We identified the runtime based on the log at the end of the first step: Cloud runtime: 1 # before
Cloud runtime: 2 # now
It was clearly stated that there won't be a decrease in compute for 1x steps Important: We want to emphasise that this is NOT a reduction in compute for 1x steps. In fact, 2x steps will see a significant INCREASE in compute allocation. The move to a dedicated CPU model means each visible CPU will provide more actual compute power. Is there something to tweak to get the same results as before ? We were already impacted by the decision to restrict the `atlassian-ip-ranges` to 4x/8x steps which forced us to move certain steps to more expensive sizes. With those two problems, our build minutes usage has significantly increased, and our bill too. Are there plans to mitigate the price increase for customers caused by this migration ? Thank you

Edmund Munday · Answer

Hi @Kirsty Mullen - apologies for the delayed response to your message. We've identified a performance regression specifically in some node environments that do very high volumes of read/write operations to disk.

We have identified a fix for this issue and are in the process of implementing that now.

Kirsty Mullen_6ghs00 · Answer

We're experiencing the same across our pipelines. It's very noticable in our vitest step (~1-2min to ~5min). Our PR pipeline has doubled in its average time overall (~6 to ~12).

Local performance is unchanged and we cannot attribute it to any code change. The cloud runtime log does indeed change on the first affected pipeline to 2.

I'll create a support ticket as advised but wanted to echo what has been raised as it's not an isolated occurrence.

Kind regards,

Edmund Munday · Answer

If anyone else is seeing measurable performance degradations of this kind of scale, please reach out to the Bitbucket Support team so that the engineering team can troubleshoot the underlying cause.

We do expect some customer to see a change in build speeds, but not an overall degradation.

The reason for this is we have moved to a much more encapsulated model when it comes to assigning CPU resources to each build, meaning builds should be more consistent now than in the past.

This means that some of your builds in the past may have been marginally faster, but an equivalent amount would have been marginally slower, leading to an average build time that is effectively equal.

For context - we conducted extensive performance testing and tuning of the new runtime over the past 3 months covers several million builds to ensure that performance was tuned to provide consistent performance.