Upgrade from 5.0.3 to 5.10.8 or 6.x Stalls out

[Kyle Lang]

When attempting an upgrade from 5.0.3 to 5.10.8 or to 6.x the upgrade stalls without error and never continues. The step it stalls on is 'Updating CONTENT table'. I've been updating a 3.2.1 environment and it's been somewhat smooth so far.

Environment: RHEL 6.7, Java 1.8.0_144, ORACLE 12C, all custom user plugins disabled and uninstalled, plugin cache deleted.

For the life of me I can't identify why the upgrade simply stops. Our DBA investigated the instance and there are no running queries found shortly after the upgrade stalls. 

2017-10-25 16:46:56,495 INFO [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [confluence.upgrade.upgradetask.UserKeyReferenceUpgradeTask] updateUsernameReferences Updating logininfo table
2017-10-25 16:46:56,552 INFO [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [confluence.upgrade.upgradetask.UserKeyReferenceUpgradeTask] updateUsernameReferences Updated 730 rows in logininfo table
2017-10-25 16:46:56,552 INFO [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [confluence.upgrade.upgradetask.UserKeyReferenceUpgradeTask] updateUsernameReferences Updating LABEL table
2017-10-25 16:46:56,706 INFO [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [confluence.upgrade.upgradetask.UserKeyReferenceUpgradeTask] updateUsernameReferences Updated 5559 rows in LABEL table
2017-10-25 16:46:56,706 INFO [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [confluence.upgrade.upgradetask.UserKeyReferenceUpgradeTask] updateUsernameReferences Updating CONTENT table

[Kyle Lang · Accepted solution]

Turns out I had multiple ORACLE drivers deployed. We were on 11G and upgraded to 12C and I had copied both the ojdbc6 and 7 drivers to the library directory. Once I removed the ojdbc6 driver the upgrade proceeded.

[Shannon S]

Thank you for providing your resolution, Kyle!

Happy you were able to get upgraded.

Take care, and Kind regards

Shannon

[Shannon S]

Hi Kyle,

Could you have a look at the Confluence logs during the upgrade and see if you notice anything unusual? 

Please feel free to share any errors you see and we can help out and determine why it's stalling.

Kind Regards,
Shannon

[Kyle Lang]

Hi Shannon,

       Working with support they discovered the ojdbc6 jar was being loaded even though I also had the ojdbc7 jar file in WEB-INF/lib. As we are on ORACLE 12 this caused the stalling. I was instructed to have only the appropriated ORACLE driver in the library directory. Once removed the upgrade progressed as intended.

It was also recommended my upgrade path be 5.0.3 -> 5.10.8 -> 6.x whereas I attempted to go straight from 5.0.3 to 6.x in my earlier testing.

[Shannon S]

Hi Kyle,

Thank you for sharing. So the issue for you is now resolved? Once you removed the other drivers and followed the intended upgrade path, everything worked out, correct?

Kind Regards,
Shannon

[Kyle Lang]

Hi Shannon,

     The issue I posted was resolved yes, but my environment doesn't work yet. Once the upgrade completes the Confluence service is immediately stopped. Confluence doesn't cleanly stop and I have to kill the java process. Once I start Confluence the upgrade steps simply run again and I have an endless loop.

The latest error I could find is below.

2017-10-27 15:53:56,432 WARN [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [hibernate.tool.hbm2ddl.PreExistingIndexChecker] warnIfExistingMatchingIndexesPresent Pre-existing indexes [CONTENT_IDX9] are present on table 'CONTENT' for columns [content_status]. New index 'c_status_idx' will be a duplicate.
2017-10-27 15:54:11,563 ERROR [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [hibernate.tool.hbm2ddl.SchemaUpdate] execute Unsuccessful: create index c_status_idx on CONTENT (CONTENT_STATUS)
2017-10-27 15:54:11,563 ERROR [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [hibernate.tool.hbm2ddl.SchemaUpdate] execute ORA-01408: such column list already indexed

2017-10-27 15:54:11,564 ERROR [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [hibernate.tool.hbm2ddl.SchemaUpdate] execute could not complete schema update
java.sql.SQLException: ORA-01408: such column list already indexed

2017-10-27 15:54:11,583 ERROR [ftbgwiiewiki301-dev01.ent.tds.net-startStop-1] [atlassian.confluence.plugin.PluginFrameworkContextListener] launchUpgrades Upgrade failed, application will not start: com.atlassian.config.ConfigurationException: Cannot update schema

[Shannon S]

Apologies for not responding on this! It looks like it missed my inbox. Happy you were able to resolve it in the end.

Shannon