Unable to start up Confluence - "Fatal error during startup sequence"

[Luke Davis]

When starting up Confluence 7.2.1 on Windows Server 2019 using MySQL 5.7, the web page displays "Confluence had problems starting up" followed by:

"Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:mananagedjobs (Start and stop the Managed Scheduled Jobs)"

"could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet"

I had to shutdown the server hosting Confluence over the weekend due to a scheduled power outage, I received the above errors when trying to start Confluence back up on Monday morning.  No other changes have been made to the server or to Confluence.  My Jira Core installation is functioning fine.

Also, not sure this is applicable, but when starting confluence from the command prompt I see this:

"---------------------------------------------------------------------------"
"Using Java: D:\jre1.8.0_231\bin\java.exe"
Error: Unable to access jarfile C:\Program
Error: Unable to access jarfile C:\Program
"---------------------------------------------------------------------------"

From the atlassian-confluence.log file:

2020-07-11 20:26:59,061 ERROR [Caesium-1-2] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Table 'confluence.scheduler_run_details' doesn't exist
2020-07-11 20:26:59,062 ERROR [Caesium-1-2] [org.hibernate.internal.ExceptionMapperStandardImpl] mapManagedFlushFailure HHH000346: Error during managed flush [org.hibernate.exception.SQLGrammarException: could not execute statement]
2020-07-11 20:26:59,062 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doCommit Commit failed. Rolling back. Error: Hibernate operation: could not execute statement; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluence.scheduler_run_details' doesn't exist
2020-07-11 20:26:59,062 WARN [Caesium-1-2] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. 

Any assistance pointing me in the right direction to get this back up and running is greatly appreciated!

[Shannon S · Accepted solution]

Hello Luke,

Thank you for raising this with us. So you confirm that you've been running on Confluence 7.2.1 for a while now, without any issues, until shutting down Confluence since last weekend. Is that right? There haven't been any config changes made in the past, perhaps that didn't apply correctly until this most recent reboot?

Generally, this is related to a collation issue, so you could use Daniel's advice in the thread below just to rule that out. I would recommend running through the steps to check your collation, even if your database was working fine with the current configuration. 

• Errror after update confluence to latest version 

Otherwise, it could be that an expired license is being used, or that there is an issue with the database name. You can review the steps in this article following the steps in Daniel's answer:

• Unknown Column Error when upgrading from Confluence 5.x to newer versions 

If you continue to have the problem, please let me know, and I can advise you further.

Take care,

Shannon

[Luke Davis]

Shannon,

Thank you for the reply.  Yes, I have been running 7.2.1 since January 2020.  No changes have been made since then.  I have checked on collation and it is set properly.  Also, we have a year license that we bought in January 2020 so we are still set there.

I noticed that the MySql .err log was over 1.3GB - I opened it and noticed the following errors dating back to March 24 2020:

2020-03-24T03:00:09.164786Z 29679 [ERROR] InnoDB: Table `confluence`.`scheduler_run_details` is corrupt.

2020-03-24T08:58:22.270675Z 41908 [Note] InnoDB: Table `confluence`.`scheduler_run_details` is corrupted. Please drop the table and recreate it

2020-03-24T08:58:22.274380Z 41908 [Warning] InnoDB: Cannot open table confluence/scheduler_run_details from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.

Running MySQLCheck --all-databases -u root -p yields:

confluence.scheduler_run_details
Warning : InnoDB: Index PRIMARY is marked as corrupted
Warning : InnoDB: Index job_id_start_time_idx is marked as corrupted
Warning : InnoDB: Index job_id_idx is marked as corrupted
Warning : InnoDB: Index start_time_outcome_idx is marked as corrupted
Warning : InnoDB: Index start_time_idx is marked as corrupted
error : Corrupt

All other tables are OK.

Unfortunately my backups only go back about a month, so I cannot restore from backup.

Can I just drop table "confluence.scheduler_run_details" and re-create it somehow?

Thanks

[Shannon S]

Hi Luke,

Have you first tried InnoDB recovery?

• Forcing InnoDB Recovery 

If so, and if that hasn't worked, I found a ticket in our support system where a user had a similar issue. The issue was with the same table, and they did drop it, but they performed the following steps first:

1. Shutdown MySQL and Confluence.
2. Started MySQL with the innodb_force_recovery = 2 switched in my.cnf. (In the article I sent you, recovery level 1 is recommended, but in this customer's case, that didn't work, and 2 was what fixed it.)
3. Confirm that MySQL is running.
4. Create full MySQL dump.
5. Then, create a MySQL dump for the individual table. (confluence.scheduler_run_details)
6. Once you confirm that the MySQL dumps have been successfully created, then you can try to drop the confluence.scheduler_run_details table.
7. Remove the innodb_force_recovery = 2 switch from my.cnf.
8. Restart MySQL.
9. Confirm MySQL is running.
10. Re-import the confluence.scheduler_run_details table from the dump file.
11. Restart Confluence.
12. Confirm Confluence is functioning.

Can you let me know if these steps help you? This isn't quite a supported workaround, as ideally you'll want to restore from a Confluence site backup, but it's worth the try to recover your site. You just want to make sure to take database dumps and a backup now of your data before making any changes.

I hope this helps!

Shannon 

[Luke Davis]

Shannon,

Those steps worked perfectly without any data loss!  Thank you so much!

Luke

[Shannon S]

Hi Luke,

That's great, I'm really glad everything ran so smoothly!

Take care, and have a pleasant rest of your week.

Shannon

[Srinivas Patruni]

HI @Shannon S I am facing similar exception when I try to upgrade Confluence from 5.8 to 6.13. It is strange that sometimes on Sandbox it is upgrading.  I opened case with Atlassian support but they could not provide a solution yet. Is this something you can help?

{noformat}

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluencedb.scheduler_run_details' doesn't exist
at sun.reflect.GeneratedConstructorAccessor387.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
at com.mysql.jdbc.Util.getInstance(Util.java:387)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:941)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3806)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2470)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2617)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:379)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:71)
... 78 more
2020-08-03 22:56:44,495 WARN [Catalina-utility-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJobDao.delete]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT (Session #1633995134)
2020-08-03 22:56:44,497 WARN [Catalina-utility-1] [impl.schedule.caesium.SchedulerClusteredJobDaoProxyWrapper] delete Could not delete clustered job 'DefaultPluginScheduler:com.atlassian.analytics.client.upload.S3EventUploader:job'
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [DELETE FROM scheduler_clustered_jobs where job_id = ?]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'confluencedb.scheduler_clustered_jobs' doesn't exist

{noformat}

[Shannon S]

Srinivas,

I would recommend running the same fix that Luke did for the table that's missing (confluencedb.scheduler_clustered_jobs) - just make sure you take a backup of everything before you make any changes. It's best to try on a staging instance first.

Otherwise, I found the case you're referring to. It looks that you raised this a while ago, but we hadn't heard from you for a few weeks, so the case closed due to timeout. I can confirm you've reopened it today, so someone from the Support team should be able to help you out soon. 

Take care,

Shannon

[Luke Davis]

I was able to correct the command prompt error "Error: Unable to access jarfile C:\Program" by creating a Windows environment variable %CATALINA_HOME% and setting it to "C:\Progra~1\Atlassian\Confluence".

However, I am still getting the same "Confluence had problems starting up" errors listed at the top of my original post.