Starting Bamboo kills Confluence

[Aram Kocharyan]

This is quite odd. I installed Bamboo using this guide:

https://confluence.atlassian.com/display/ATLAS/Dragons+Stage+8+-+Install+Bamboo

Afterwards, I modified the ports as it asked and started up Bamboo. All was well, then I checked my Confluence and saw it failed to load with this message:

503: Service Temporarily Unavailable

I then noticed that the Confluence java process (tomcat) had been killed. I searched through which files had changed:

[root@swen90013-C bamboo]# find /var/atlassian/application-data/confluence/ -mtime -0.1
/var/atlassian/application-data/confluence/logs/atlassian-confluence.log
/var/atlassian/application-data/confluence/index/didyoumean
[root@swen90013-C bamboo]#

[root@swen90013-C aramk]# find /opt/atlassian/confluence/ -mtime -1
/opt/atlassian/confluence/logs
/opt/atlassian/confluence/logs/localhost.2013-04-02.log
/opt/atlassian/confluence/logs/catalina.out
/opt/atlassian/confluence/logs/catalina.2013-04-02.log
/opt/atlassian/confluence/logs/host-manager.2013-04-02.log
/opt/atlassian/confluence/logs/manager.2013-04-02.log
/opt/atlassian/confluence/conf
/opt/atlassian/confluence/temp
/opt/atlassian/confluence/temp/+~JF5539691965213085257.tmp
/opt/atlassian/confluence/temp/+~JF1483396561054584346.tmp
/opt/atlassian/confluence/temp/+~JF3629011383201427068.tmp
/opt/atlassian/confluence/temp/+~JF355316828638181202.tmp
/opt/atlassian/confluence/temp/+~JF518905120637013707.tmp
/opt/atlassian/confluence/temp/+~JF720759182393898308.tmp
/opt/atlassian/confluence/temp/+~JF8465892025386154866.tmp
/opt/atlassian/confluence/temp/+~JF410444077936750596.tmp
/opt/atlassian/confluence/work
/opt/atlassian/confluence/work/catalina.pid

[root@swen90013-C aramk]# find /opt/bamboo/ -type f -mtime -0.1
/opt/bamboo/logs/bamboo.log
/opt/bamboo/logs/atlassian-bamboo.log
/opt/bamboo/jms-store/bamboo/KahaDB/db.data
/opt/bamboo/jms-store/bamboo/KahaDB/db-1.log
/opt/bamboo/jms-store/bamboo/KahaDB/db.redo

Unfortunately, the confluence log shows no indication of failure - during the time of the halt there is nothing in atlassian-confluence.log. The same applies to the confluence catalina.out.

So from what I understand, the process was forcefully terminated at some point. The only error in the bamboo logs is this:

2013-04-02 19:54:53,834 WARN [main] [BrokerService] Store limit is 102400 mb, whilst the data directory: /opt/bamboo/jms-store/bamboo/KahaDB only has 789 mb of usable space
2013-04-02 19:54:53,834 ERROR [main] [BrokerService] Temporary Store limit is 51200 mb, whilst the temporary data directory: /opt/bamboo/jms-store/bamboo/tmp_storage only has 789 mb of usable space

If I start confluence, it seems to kill JIRA! And if I start JIRA it kills Bamboo...

Is it the low memory in my VM?

[root@swen90013-C aramk]# free
             total       used       free     shared    buffers     cached
Mem:       1922632    1834332      88300          0      14984      39252
-/+ buffers/cache:    1780096     142536
Swap:      1268728     941300     327428

What's going on??

[Andy Brook [Plugin People] · Accepted solution]

you may well be running out of heap. One solution is to grab all required memory for each java process, do this by setting the Minimum Heap and the Maximum Heap to be the same JAVA_OPTS value in the various setenv.sh files. You will then be unable to launch new processes if your OS has insufficient RAM to run them, rather than they dyning randomly depending on load.

eg:

-Xms1024m -Xmx1024m

[Aram Kocharyan]

Thanks Andy. So you suspect it's the RAM? We've got 2GB on our uni VM but can ask for more if needed. We're running like 4 tomcat instances so it's pushing the RAM to the max. Perhaps allocating more swap will help, at least to prevent it from crashing?

[Andy Brook [Plugin People]]

Yes, if the OS has swap that will work, but it will run like a dog :) If you are doing a POC , go light, perhaps you can get away with -xms512m -xmx512m, or lower, barely. Use jconsole or jvisualVM to check actual heap usage, you can also tune permgen this way. If you are running on a remote VM, tunnel X with 'ssh -X' to allow jconsole to be executed remotely.

[Aram Kocharyan]

Cheers for the tips. I think we're essentially running out of resources, so I'll look into getting an upgrade. Glad to know it's not a configuration issue :)