Pipeline fails with "Container 'Build' exceeded memory limit." during Build teardown

[Jay Seletz]

We have a pipeline running a gradle build that runs a lot of Junit tests as well as Jest unit tests.  When all of the tests pass, the build succeeds.  When there is even one test failure, the build test fails, but completes.  Then the "Build teardown" starts and seems to abort partway through.  The pipeline shows the "Container 'Build' exceeded memory limit." error, and no artifacts are uploaded in this case (typically the test results are uploaded as artifacts even when there are failing tests causing the build step to fail).  The pipeline is running with 2x memory.   

It looks like the Build teardown is what is exceeding container memory.  Does the teardown step also run in the "build" container?  Is it possible that the test reports being saved as an artifacts are causing the memory to be exceeded?  Since teardown is a separate step, is that running in it's own build container instance?  How much memory does that step get?

[Patrik S]

Hello @Jay Seletz and thank you for reaching out to Atlassian Community!

The Build teardown step also runs in the build container, the same container where your step's script runs, so it will have the same amount of memory you configured for the build container (step memory minus any services you using). For example, if you have a step with 2x with a database service, the step will receive 8GB of memory, being 1GB reserved for the database service container, and the remaining 7GB assigned to the build container.

As you mentioned the memory exceeding error just occurs when there are failing tests, I suspect your build is running very close to the memory limit and once it reaches the build teardown phase it's already out of memory causing that step to abort. 

Since you mentioned using Jest framework to execute some of your tests, we have seen cases in the past where builds using this framework would get stuck or fail with memory exceeded error ( see Scenario 2.2: Builds using Jest Test Framework are slow or frequently hang (based on the Pipeline build minutes consumption) or failed with Container “Build” exceeded memory limit error). In this case, I would suggest adding the following flags to your Jest command and try running it again :

jest --runInBand --maxWorkers=2

This is to limit the number of workers created by Jest to execute the tests. By default, Jest creates 7 child workers, and with the command above we are reducing to 2 workers to reduce the memory usage.

If that still does not work, you can try to debug your pipeline locally with docker to identify how much memory it's using and optimize it accordingly. If you need more than 8GB for your build, you have the option to use linux docker self-hosted runners, where the step will run in your infrastructure and can be configured to use up to 32GB of memory.

Thank you, @Jay Seletz !

Patrik S