Bamboo won't start after upgrade and system restart

[Masterfalcon]

After recently upgrading to Bamboo 4.4.4 everything was running smoothly. A couple days later after a system reboot it no longer starts up. There are complaints about duplicate keys in the log and when I try to access the system I get a HTTP 503 error.

Attached is my log. Hopefully it may help shed some light on what's going on and if there's a way to resolve the problem.

Thanks!

(bamboo.log)

After looking some more, this seems to be the core issue and I have no idea what to do about the unique constraint violation:

2013-04-02 23:29:33,675 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2] [BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@309b3e5e -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:

java.sql.SQLException: error in script file line: 7379 org.hsqldb.HsqlException: Violation of unique constraint $$: duplicate value(s) for column(s) $$: SYS_PK_75

at org.hsqldb.jdbc.Util.sqlException(Unknown Source)

at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)

at org.hsqldb.jdbcDriver.getConnection(Unknown Source)

at org.hsqldb.jdbcDriver.connect(Unknown Source)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

[Masterfalcon]

I found that I do have a defaultdb.script file that appears to contain SQL commands for my latest good db. Is it possible to use that to create a new database, export from Bamboo and then create a new instance from that data?

[danielwester]

Looking at the log file - it looks like you're running hsqldb which would create the db in your bamboo home ( from your log: /Volumes/Drobo/bamboo ). I would suggest making sure that's mountable and that you can see /Volumes/Drobo/bamboo/database/defaultdb . If you're using this Bamboo instance for a production instance - you should probably switch to an external database.

[Masterfalcon]

Thanks Daniel,

The drive is mounted but the database file does not exist.

[Masterfalcon]

I found that I do have a defaultdb.script file that appears to contain SQL commands for my latest good db. Is it possible to use that to create a new database, export from Bamboo and then create a new instance from that data?

[Septa Cahyadiputra]

Hi David,

Judging from your logs, it seems that Bamboo is having an issue opening connection to your database. Could you please double check if Bamboo connection to the database is working fine and please double check if the database is alive and healthy to server Bamboo.

Caused by: 
java.sql.SQLException: Connections could not be acquired from the underlying database!

Hope it helps.

Cheers,
Septa Cahyadiputra

[Masterfalcon]

How do i do that? I am using whatever is set up as the default database type.

[Masterfalcon]

I checked again and indeed, the database that it is trying to connect to does not exist but I have no idea where it may have gone.

[Septa Cahyadiputra]

Hi David,

Seems like data corruption might be the cause of the issue as HSQLDB is a memory based database that intended to use only for evaluation purposes. To avoid such occurrences in the future, you might want to migrate your database to an external database, which is much more reliable.

Cheers,
Septa Cahyadiputra