In our project we have a Bamboo configuration which runs on EC2 instance with Default Image EBS x86_64 (linux). Under this configuration we have Maven build with a continuous Java test. This test may last for more than one day and produces a lot of log data which by default goes to the system output stream (which is then redirected to ErrorStreamToBuildLoggerOutputHandler by Bamboo). It is about 20 log entries per second.
After ~12 hours we see on Atlassian Bamboo page that the build is marked with unknown status and Build Result Summary reports Reason for not building is unknown. As a result EC2 instance is shut down and our test does not complete. That is the main problem. There are no signs of errors in the test itself.
We investigated deeper, ssh-logined into this EC2 instance and monitored Java processes and logs. Normally there are three Java processes running (two for Bamboo and one for our Maven). We found that there happens an OOM error in the Bamboo Java process (see bamboo_problematic_process.txt attached). You can see the relevant part of bamboo-elastic-agent.out attached.
Also a heap dump was created (see a small screenshot from MAT attached). In this dump we see a huge (~170MB) object of type com.atlassian.bamboo.build.logger.interceptors.ErrorMemorisingInterceptor which contains ~1 million of com.atlassian.bamboo.build.ErrorLogEntry objects. They seem to be all entries once sent to ErrorStreamToBuildLoggerOutputHandler.
After OOM in Bamboo JVM we see some deadlocks in it and also the Maven process seems to freeze.
We try to workaround the issue by configuring the logging in our test. But still there seems to be an issue with Bamboo logging/memory (default) settings. We found no way to change these settings for OnDemand Bamboo.
