Database access problem

[Mark Ruedy]

After a restart I am unable to bring stash back up.

Stash version 3.8.0

Database PostgreSQL: 8.4.20

Below is the top of the log errors.  I don't see how to debug this.  BTW, I only have a standalone stash server, I don't know why the error mentions clusterSchedule

How do I debug this?

I have tried restarting postgresql too and it's tables and roles look right.

2015-04-24 13:22:23,091 ERROR [clusterScheduler_Worker-5] com.jolbox.bonecp.ConnectionHandle Database access problem. Killing off all remaining connections in the connection pool. SQL State = 57P01
2015-04-24 13:22:23,133 ERROR [clusterScheduler_Worker-5] c.a.scheduler.core.JobLauncher Scheduled job with ID 'BatchNotificationJobRunner' failed
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
Database:
- name:PostgreSQL
- version:8.4.20
- minor version:4
- major version:8
Driver:
- name:PostgreSQL Native Driver
- version:PostgreSQL 9.3 JDBC4.1 (build 1102)

org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:161) ~[na:na]

[Dop Sun]

See the same issue while installing using MySql with latest version of Stash. But it runs fine afterwards so far.

2015-07-09 23:04:21,248 ERROR [clusterScheduler_Worker-6]  c.a.scheduler.core.JobLauncher Scheduled job with ID 'BatchNotificationJobRunner' failed

com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:

Database:

 - name:MySQL

 - version:5.1.63-0ubuntu0.10.04.1

 - minor version:1

 - major version:5

Driver:

 - name:MySQL Connector Java

 - version:mysql-connector-java-5.1.33 ( Revision: alexander.soklakov@oracle.com-20140908134200-8ukofe1izi0r2b63 )

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'stashdb.AO_92D5D5_USER_NOTIFICATION' doesn't exist

 at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:161) ~[na:na]

 at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects.find(TenantAwareActiveObjects.java:323) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.dao.AoUserNotificationDao.findAll(AoUserNotificationDao.java:41) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.BatchNotificationJob.processWithTime(BatchNotificationJob.java:121) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.BatchNotificationJob.processInTimer(BatchNotificationJob.java:100) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.BatchNotificationJob.process(BatchNotificationJob.java:93) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.BatchNotificationJob.process(BatchNotificationJob.java:80) ~[na:na]

 at com.atlassian.stash.internal.notification.batch.BatchNotificationJob$BatchNotificationJobRunner.runJob(BatchNotificationJob.java:256) ~[na:na]

 at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153) [atlassian-scheduler-core-1.6.0.jar:na]

 at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118) [atlassian-scheduler-core-1.6.0.jar:na]

 at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97) [atlassian-scheduler-core-1.6.0.jar:na]

 at com.atlassian.scheduler.quartz2.Quartz2Job.execute(Quartz2Job.java:32) [atlassian-scheduler-quartz2-1.6.0.jar:na]

 at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:na]

 at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:na]

 ... 21 frames trimmed

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'stashdb.AO_92D5D5_USER_NOTIFICATION' doesn't exist

 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_25]

 at com.mysql.jdbc.Util.handleNewInstance(Util.java:377) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.Util.getInstance(Util.java:360) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030) ~[mysql-connector-java-5.1.33-bin.jar:5.1.33]

 at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:172) ~[bonecp-spring-0.7.1.RELEASE.jar:0.7.1.RELEASE]

 at net.java.ao.EntityManager.find(EntityManager.java:765) ~[na:na]

 at net.java.ao.EntityManager.find(EntityManager.java:675) ~[na:na]

 at net.java.ao.EntityManager.find(EntityManager.java:584) ~[na:na]

 at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.find(EntityManagedActiveObjects.java:157) ~[na:na]

 ... 14 common frames omitted

[ThiagoBomfim]

Hi Mark,

Can you make sure your database is up and running? From the page below:

• http://www.postgresql.org/docs/8.0/static/errcodes-appendix.html

Looks like your database was brought down. I believe that it was still coming back up and that's why this error on the Stash logs.

Best,
Thiago

[Mark Ruedy]

Yes, that was it. I was sure I had restarted it before I posted this but then afterward I tried again and it worked fine. Thanks

[ThiagoBomfim]

Hi Mark, I am glad that helped you! Best, Thiago.

[Boris Berenberg]

Hey Mark,

This is definitely an issue where we will need to see your logs. Take a look at https://confluence.atlassian.com/display/STASH/Raising+a+request+with+Atlassian+Support