As part of the roll-out of our new V2 Pipelines runtime, the Bitbucket Pipelines team is gradually migrating customers across to the new environment to ensure we can monitor any issues and provide support to customers as required.
During this process, a small number of issues have been identified with builds utilising deprecated/out of support toolchain components. This page will provide an up to date list of all identified issues, along with suggestions on how to identify and troubleshoot those problems.
Out of Memory:
This class of error relates to builds running out of memory where they were previously stable.
This is experienced as steps exceeding the previously assigned memory boundaries and failing.
Most Out Of Memory errors we observe are not related to this change, and may just be a result of resource requirements changing, please attempt to rule out any other changes first.
For those cases that are related to the runtime change, the main cause of these errors appears to be legacy tools or runtimes which are not cgroupv2 aware or are dependent on cgroupv1 functionality to define resource boundaries.
Our runtime v2 along with all major linux distributions have now standardised on cgroupsv2.
https://kubernetes.io/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/
We do not recommend depending on cgroups as a resource control mechanism; configuring explicit resource boundaries around your tooling is a preferential approach to ensure deterministic builds.
Mitigation options:
Upgrade your tooling to cgroupsv2 aware versions, most software vendors have back-ported patches, even to very old versions such as Open JDK 8
Configure your tooling to utilise explicitly defined limits, e.g. java -Xms512m -Xmx2048
Increase your step size if you require more resources.
If you’re unable to perform those tasks, please raise a support ticket, a temporary exemption may be possible, but will not be permanent.
We will add to this page if any further issues are identified.
Thanks to those users who supported us during the initial roll-out, your patience and assistance has enabled us to continue to make our product more consistent, reliable and performant.
Liam Nunns
3 comments