Bamboo server hang during startup

[Chris Hammond]

Bamboo had a job that was stuck in the "Queued" state, but did not show up in the list of Builds. So we restarted Bamboo. Upon restart, the catalina log file was stuck on a line that said "Initializing C3P0 pool". After a while, an exception was thrown that said "The database is in use by another process". There were no processes running on the machine owned by me, other than my login and tomcat. Could the default database be corrupted or locked?

I found an article that said to delete the KahaDB and caches directories from the bamboo home directory to solve this problem

I tried renaming them to caches_old and KahaDB_old and restarting the tomcat server

The same thing happened again with the hang on Initializing C3P0 and subsequent crash

[Daniel Wester]

You created your jira issue in the jira.atlassian.com. If you want support - create an issue at support.atlassian.com Secondly if you're using bamboo for non eval purposes- get off the embedded hsql db and move to a proper external db. Thirdly move the bamboo home dir locally. There are only a couple of dirs and files you'll need to mount on NFS (or rsync) the artifacts and the bamboo.cfg.xml are 2 of them, I think there are a couple of others). You don't want git to be checking out into a NFS dir (or to be doing builds in it)

[Przemek Bruski]

You're using the embedded HSQLDB database and the previous Bamboo process is still running. Try killing it forcibly.

[Chris Hammond]

Also, here is the JIRA issue that I logged with the atached catalina.out from Tomcat. https://jira.atlassian.com/browse/TST-54941

[Chris Hammond]

As I mentioned, I did kill it forcibly with a "kill -9". I then checked with a "ps -eaf" and it definitely was not running.

[Przemek Bruski]

Is Bamboo home on an NFS mount?

[Chris Hammond]

I need to head out now, and will reply to any other insights you might have when I get back

[Chris Hammond]

It is. I thought that might be a problem as well, but I checked and do not see other instances of bamboo running off of that directory on other machines. The only person that starts and stops bamboo is me, and I definitely did not start it up on another machine. It was working and I simply restarted it.

[Przemek Bruski]

NFS v3, probably? Lock support is handled by an external process (not NFS core) and it still thinks that the files are locked. I can't recommend that setup. At the very least, upgrade to NFS v4.

I'd also recommend symlinking the database dir and the JMS dir to local dirs if you absolutely must run on NFS.