OnDemand Elastic Bamboo log entries causing OutOfMemory

In our project we have a Bamboo configuration which runs on EC2 instance with Default Image EBS x86_64 (linux). Under this configuration we have Maven build with a continuous Java test. This test may last for more than one day and produces a lot of log data which by default goes to the system output stream (which is then redirected to ErrorStreamToBuildLoggerOutputHandler by Bamboo). It is about 20 log entries per second.

After ~12 hours we see on Atlassian Bamboo page that the build is marked with unknown status and Build Result Summary reports Reason for not building is unknown. As a result EC2 instance is shut down and our test does not complete. That is the main problem. There are no signs of errors in the test itself.

We investigated deeper, ssh-logined into this EC2 instance and monitored Java processes and logs. Normally there are three Java processes running (two for Bamboo and one for our Maven). We found that there happens an OOM error in the Bamboo Java process (see bamboo_problematic_process.txt attached). You can see the relevant part of bamboo-elastic-agent.out attached.

Also a heap dump was created (see a small screenshot from MAT attached). In this dump we see a huge (~170MB) object of type com.atlassian.bamboo.build.logger.interceptors.ErrorMemorisingInterceptor which contains ~1 million of com.atlassian.bamboo.build.ErrorLogEntry objects. They seem to be all entries once sent to ErrorStreamToBuildLoggerOutputHandler.

After OOM in Bamboo JVM we see some deadlocks in it and also the Maven process seems to freeze.

We try to workaround the issue by configuring the logging in our test. But still there seems to be an issue with Bamboo logging/memory (default) settings. We found no way to change these settings for OnDemand Bamboo.

2 answers

Hi Alexander,

Please raise a support ticket on https://support.atlassian.com, since here is a forum platform, so we cannot provide you a more detailed support. We need to proceed with a heap dump analisys, so please attach it to the created ticket.


Lucas Lima

Atlassian Support

As a workaround for this, try suppressing as much logging as you can for your tests. I added a seperate log4j.properties file and a seperate hibernate.cfg.xml file in my test directory that was less verbose. That got my build working finally without running out of heap.

Suggest an answer

Log in or Join to answer
Community showcase
Louis De Jaeger
Posted yesterday in Off-topic

Friday fun: your best joke

Hi all Lets make this Friday fun really fun and post one (or more) of your best jokes! The joke can be about an Atlassian product, or just a really fun joke you want to share! I’m not the best j...

158 views 12 3
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot