Error 404 after upgrading confluence from 5.2.3 to 6.2.1

[Sandra Langer]

Long term goal: migrate fom 5.2.3 on MS SQL Server to 6.2.1 on centos/postgresql.

My migration plan:

1. bring up test environment with 5.2.3 on internal db as http://localhost:8990 -> went fine
2. export site from production to xml
3. import site to test 5.2.3 on internal db and test on http://localhost:8990 -> went fine
4. upgrade test to 6.2.1 on internal db -> I'm stuck here
5. export site from test 6.2.1 -> to be done
6. import site to prod 6.2.1 on centos/postgresql -> to be done

On step 4, after upgrading my test environment from 5.2.3 to 6.2.1 (by running the installer and select upgrade existing installation), I get HTTP ERROR 404 on http://localhost:8090. Restarting service or entire machine does not help.

catalina.log says (besides some warnings and infos):

06-Jun-2017 22:18:40.637 SCHWERWIEGEND [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
06-Jun-2017 22:18:40.637 SCHWERWIEGEND [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
06-Jun-2017 22:19:05.621 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jun-2017 22:19:05.622 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jun-2017 22:19:05.622 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jun-2017 22:19:05.622 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jun-2017 22:19:05.622 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.hsqldb.jdbc.JDBCDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
06-Jun-2017 22:19:05.623 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.624 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.624 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.625 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.625 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.626 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.627 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [scheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
06-Jun-2017 22:19:05.627 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [HSQLDB Timer @6388b21] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)
org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
java.lang.Thread.run(Thread.java:748)
06-Jun-2017 22:19:05.628 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
06-Jun-2017 22:19:05.628 WARNUNG [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [AtlassianEvent::CustomizableThreadFactory-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)

atlassian-confluence.log says (beside many more):

2017-06-06 22:15:23,423 ERROR [localhost-startStop-1] [atlassian.confluence.plugin.BandanaPluginStateStore] load Exception when loading plugins state from the database. Falling back to in memory state.
java.util.concurrent.CompletionException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tenantedTransactionManager' defined in class path resource [databaseSubsystemContext.xml]: Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'tenantedLocalSessionFactoryBean5' defined in class path resource [productionDatabaseContext.xml]: Invocation of init method failed; nested exception is org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]

I assume something went wrong with internal database configuration/Upgrade. Any hints how to resolve this?

[Marcelo Horlle · Accepted solution]

Hey Sandra,

Based on the amount of SQL errors on yours logs, I'd do the plan slightly different, since I believe that migrating from an external database to an internal database won't bring you good results. In order to provide root cause, ideally, you would need to create a ticket with support and attach a Support Zip, which contain more logs. As this is not a ticket, here is what I suggest you to do, since you are using test instances, which is great:

1. Create a Test Environment with CentOS/PostgreSQL.

2. Install Confluence 5.2.3 on this test environment.

3. Export Production site as XML.

4. Import the Production XML to the test environment.

5. Run all the needed tests to ensure that the new database/OS works fine.

6. Upgrade your test environment to Confluecne 6.x version.

7. If all is good, these are great news! Update your production environment to CentOS/PostgreSQL.

8. Install Confluence 6.x on Production as a fresh installation.

9. Take a XML Backup from your test 6.x Instance.

10. Import this XML Backup in Production.

This should work fine. Most than likely these errors are happening because when migrating to an External Database to Embedded Database, which is not an option on our documentation, the database connection properties from external database are overriding the internal database ones.

Hope this helps.

[Sandra Langer]

Hi Marcelo,

thank you very much for this in-depth answer :)

I will got that Path in  the next couple of weeks and report back if there were any new problems.

[Marcelo Horlle]

Hey Sandra,

That is cool! Have a great weekend ahead. :)