Pipelines memory limit exceeded

[m_t_w]

we use pipelines.

we sometimes have jobs that fail with the message 'Build memory limit exceeded.'

we also run the same docker image internally as we run in pipelines. (we run the same image locally as we declare in the image: of bitbucket-pipelines.yml)

we want a way to run our docker image locally with the same limitations that pipelines enforces (4gb) : https://confluence.atlassian.com/bitbucket/limitations-of-bitbucket-pipelines-827106051.html#LimitationsofBitbucketPipelines-Buildlimits

we want to do this to make sure we are staying within 90% of what the limit is of pipelines.  (so 3.6GB)

Is this a way to achieve that?

docker run --rm --memory=3600M --memory-swap=3600M docker-image-same-as-we-run-in-pipelines

[Philip Hodder]

Are you using any service containers or the Docker build and push functionality? This will *also* use part of your 4GB of memory for the build. (1GB per service container)

(See here: https://confluence.atlassian.com/bitbucket/use-services-and-databases-in-bitbucket-pipelines-874786688.html )

EDIT: This was a bug. It's currently being worked on. Follow it here: https://bitbucket.org/site/master/issues/14666/enabling-docker-for-pipelines-wrongfully

[zt_tasos]

Nope, it's just the maven build with tests

[James Clinton]

Same here.  Just maven with tests.  Please help us debug this.

[Philip Hodder]

Sorry for the delayed response.

Can you try a slight variation fo your command for your local debugging.

docker run -it --volume=/Users/myUserName/code/localDebugRepo:/localDebugRepo --workdir="/localDebugRepo" --memory=4g --memory-swap=4097m --entrypoint=/bin/bash python:2.7

Akin to these docs: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html

The difference is I've changed memory-swap from 4g to 4097m (purposefully 1 MB larger than 4GB).

I need to do a little more investigation, but if the values are the same then docker may revert to default swap behaviour, giving you 8GB of swap instead of 0GB. 

I'll update the docs once I confirm this. :) See if that change helps with reproducing the issue.

[Maksim Muruev]

Hi here I thinks I have the same issue but for node js build! It works pretty fine at localhost with 1gb limits.. but at build pipelinest it doesn't complains about memory limits! Event x2 cant help!

[mkucharekxo]

Hi! actually, we're still having the same issue - a simple maven build with tests is failing unless we use `size: 2x`.. We've tried limiting memory with MAVEN_OPTS, as well as setting

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap`

parameters (we're using Java 8, so technically without these flags, JVM is not aware it's running in a container), but still no luck..

Did anyone happen to find a solution for that?

[Alexander Kasper]

Same issue for us. We're doing a Scala (sbt plus ScalaTest) build using a PostgreSQL database as a service. When I run this locally, it works fine using about 1GB of memory. If I understand the documentation correctly, Posgtres will use 1GB of the 4GB for a pipeline. So it should not run out of memory?

[Alexander Kasper]

I also ran the build in Docker as suggested in the Pipelines documentation for debugging locally and the build is fine there as well with the container using about 1.6GB of memory. For some branches the memory exceeding happens only occasionally (immediate rerun mostly works), while one other branch does not work at all atm. This suggests to me that the build is very close to some kind of memory threshhold, but I do not know what is the limiting factor here? Any tips how to debug this?

[zt_tasos]

We have the same issue with our maven build.

Anybody has found a way to hard limit maven to less than 4G? 

[Luca Morettoni]

We have the same issue with our Java project; during the building process we get a "Build exceeded memory limit" error, but if we run the build locally (as reported here: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html) all works fine without any problem.

I also tried to limit the memory allocated by maven with the MAVEN_OPTS variable, but anyway the build is stopped (randomly).

[James Clinton]

Is there a fix for the memory limit issue?