Pipelines runtime v2 - rollout updates & known issues.

[#1]

As part of the roll-out of our new V2 Pipelines runtime, the Bitbucket Pipelines team is gradually migrating customers across to the new environment to ensure we can monitor any issues and provide support to customers as required.

During this process, a small number of issues have been identified with builds utilising deprecated/out of support toolchain components. This page will provide an up to date list of all identified issues, along with suggestions on how to identify and troubleshoot those problems.

Out of Memory:
This class of error relates to builds running out of memory where they were previously stable.
This is experienced as steps exceeding the previously assigned memory boundaries and failing.

Most Out Of Memory errors we observe are not related to this change, and may just be a result of resource requirements changing, please attempt to rule out any other changes first.

For those cases that are related to the runtime change, the main cause of these errors appears to be legacy tools or runtimes which are not cgroupv2 aware or are dependent on cgroupv1 functionality to define resource boundaries.
Our runtime v2 along with all major linux distributions have now standardised on cgroupsv2.

https://kubernetes.io/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/

We do not recommend depending on cgroups as a resource control mechanism; configuring explicit resource boundaries around your tooling is a preferential approach to ensure deterministic builds.

Mitigation options:

• Upgrade your tooling to cgroupsv2 aware versions, most software vendors have back-ported patches, even to very old versions such as Open JDK 8

• Configure your tooling to utilise explicitly defined limits, e.g. java -Xms512m -Xmx2048

• When utilising docker in docker, you should define limits for your child containers. e.g.
  docker run --memory 1G amazoncorretto:17.0.12 /bin/sh

• Increase your step size if you require more resources.

• If you’re unable to perform those tasks, please raise a support ticket, a temporary exemption may be possible, but will not be permanent.

We will add to this page if any further issues are identified.

Thanks to those users who supported us during the initial roll-out, your patience and assistance has enabled us to continue to make our product more consistent, reliable and performant.

[Marcos Sampaio]

Unsupported Features on Runtime v2:

Docker init process

The docker run --init command is currently not supported. If you attempt to use this parameter with docker run or specify init: true in Docker Compose, you will encounter an error message similar to the following:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/usr/local/bin/docker-init" to rootfs at "/sbin/docker-init": mount /proc/self/fd/6:/sbin/docker-init (via /proc/self/fd/10), flags: 0x1021: operation not permitted: unknown.</code>

[Ucchishta Sivaguru]

Oh wow, so this just happened recently. Thanks, @Marcos Sampaio !
When removing init: true from our docker-compose file, are there specific considerations we should keep in mind? Does the new Bitbucket Pipelines runtime handle zombie process reaping automatically, or do we need to manage this ourselves?

[ben_clifford]

For future viewers: `4x` and `8x` are new options for `size`