Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Container 'docker' exceeded memory limit. - Confusing cause / remedy

Edited

I know there is a lot out there about this, but so far I've not seen an explanation.

We intermittently have build failures with this issue. No material changes between source code or artifacts that would cause it, and re-running the build could actually succeed.

We have a "build" step that we are using size: x1 and had allocated 3072 memory to "docker".. our only service in the step.

The Docker In Docker (DinD) build line in the step (docker build ....) launches a multi layer .net 6 sdk build layer, adds node etc... and builds a c# app and react typescript front end client.

When the exceed memory happens, its always during the React-Scripts step... "Creating a production build..." and the error is "Unexpected EOF" (due to the OOM)

At first I assumed it was the DinD that was exceeding the 3072 memory it was given.

So I added the --memory="3g" and --memory-swap="3g" switches to the docker build command hoping that it would stay within its assigned params.

This works fine locally, but still not in the pipeline.

Then I started suspecting the bitbucket agent container was throwing the message not the DinD docker service...

So I started reducing the amount allocated to the DinD service to give more headroom to the bitbucket agent container... I tried docker.memory: 2048 and appropriate --memory="2g" switches, that leaves the bitbucket agent with 2048 headroom.

Still broken.

Then I decided to size: x2 without changing the DinD docker service memory allocation.

Voila!

So it was the bitbucket agent container running out of headroom... but... with what?? The DinD service should have been bounded to the memory settings it was given, so what was filling up the remaining agent container memory?

Totally different tech stack I hope, but I recall Windows Docker Desktop having issues with rampant WSL memory at one point requiring a WSL config change.

The problem with the "just x2 the size to fix it " is that its "double the minutes / double the cost"... and if we don't know why its happening, what is to say that its not a Bitbucket bug causing this? When is x2 not gonna be enough?

I would be grateful for any insight on the container memory consumption...

Thanks!

1 answer

0 votes
Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
Feb 20, 2023

Hi Chris and welcome to the community!

You can add the following commands in your bitbucket-pipelines.yml file, at the beginning of the script of the step that is failing with memory issues:

- while true; do echo "Memory usage in bytes:" && cat /sys/fs/cgroup/memory/memory.memsw.usage_in_bytes; sleep 2; done & 2
- while true; do date && ps aux && echo "" && sleep 2; done &

These commands will print memory usage per process in the Pipelines build log for the duration of the build. If you observe the output and you think there might be a bug, you can create a ticket with the support team and provide the URL of the build that includes the above commands for further investigation. You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

If you have any questions, please feel free to let me know.

Kind regards,
Theodora

Thank you! I will give that a try!

Like Theodora Boudale likes this

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events