Container 'docker' exceeded memory limit. - Confusing cause / remedy

[Chris Heale]

I know there is a lot out there about this, but so far I've not seen an explanation.

We intermittently have build failures with this issue. No material changes between source code or artifacts that would cause it, and re-running the build could actually succeed.

We have a "build" step that we are using size: x1 and had allocated 3072 memory to "docker".. our only service in the step.

The Docker In Docker (DinD) build line in the step (docker build ....) launches a multi layer .net 6 sdk build layer, adds node etc... and builds a c# app and react typescript front end client.

When the exceed memory happens, its always during the React-Scripts step... "Creating a production build..." and the error is "Unexpected EOF" (due to the OOM)

At first I assumed it was the DinD that was exceeding the 3072 memory it was given.

So I added the --memory="3g" and --memory-swap="3g" switches to the docker build command hoping that it would stay within its assigned params.

This works fine locally, but still not in the pipeline.

Then I started suspecting the bitbucket agent container was throwing the message not the DinD docker service...

So I started reducing the amount allocated to the DinD service to give more headroom to the bitbucket agent container... I tried docker.memory: 2048 and appropriate --memory="2g" switches, that leaves the bitbucket agent with 2048 headroom.

Still broken.

Then I decided to size: x2 without changing the DinD docker service memory allocation.

Voila!

So it was the bitbucket agent container running out of headroom... but... with what?? The DinD service should have been bounded to the memory settings it was given, so what was filling up the remaining agent container memory?

Totally different tech stack I hope, but I recall Windows Docker Desktop having issues with rampant WSL memory at one point requiring a WSL config change.

The problem with the "just x2 the size to fix it " is that its "double the minutes / double the cost"... and if we don't know why its happening, what is to say that its not a Bitbucket bug causing this? When is x2 not gonna be enough?

I would be grateful for any insight on the container memory consumption...

Thanks!

[Theodora Boudale]

Hi Chris and welcome to the community!

You can add the following commands in your bitbucket-pipelines.yml file, at the beginning of the script of the step that is failing with memory issues:

- while true; do echo "Memory usage in bytes:" && cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes; sleep 2; done & 2
- while true; do date && ps aux && echo "" && sleep 2; done &

These commands will print memory usage per process in the Pipelines build log for the duration of the build. If you observe the output and you think there might be a bug, you can create a ticket with the support team and provide the URL of the build that includes the above commands for further investigation. You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

If you have any questions, please feel free to let me know.

Kind regards,
Theodora

[Chris Heale]

Thank you! I will give that a try!