Unable to start confluence after upgrade

[gaurav.trivedi]

Hello, 

We have recently upgraded confluence from 6.3 to 6.7. After the upgrade, it wont start and fail with below error. 

if you can help us with this. How to resolve this? 

Updated Logs 3/29

Catalina out

10:02:05.852 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
10:02:05.852 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.43
10:02:23.823 SEVERE [Standalone-startStop-1] org.apache.catalina.core.ContainerBase.startInternal A child container failed during start
java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Standalone].StandardHost[localhost].StandardContext[]]
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:871)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Standalone].StandardHost[localhost].StandardContext[]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)
... 6 more

Conflence Log

2020-03-05 13:57:42,512 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:02:42,510 WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:07:42,510 WARN [Caesium-1-1] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:12:42,511 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:17:42,511 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:22:42,513 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:27:42,511 WARN [Caesium-1-3] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:32:42,512 WARN [Caesium-1-1] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:37:42,513 WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:42:42,514 WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:47:42,513 WARN [Caesium-1-1] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:52:33,384 WARN [http-nio-8090-exec-2] [internal.integration.jira.DefaultJiraService] communicateWithJira Problem communicating with JIRA: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
-- referer: http://localhost:8090/spaces/viewspace.action?key=HIPAA | url: /rest/highlighting/1.0/panel-items | traceId: a5047c1f57aa8ce6 | userName: alexander.korzun
2020-03-05 14:52:33,388 WARN [http-nio-8090-exec-2] [confluence.plugins.createjiracontent.CachedJiraResourcesManager] getRemoteJiraServer Could not detect issue creation supported or not in Jira server The server may be unreachable. Please ensure you are running JIRA 5.0 or higher.
-- referer: http://localhost:8090/spaces/viewspace.action?key=HIPAA | url: /rest/highlighting/1.0/panel-items | traceId: a5047c1f57aa8ce6 | userName: alexander.korzun
2020-03-05 14:52:33,412 WARN [http-nio-8090-exec-2] [internal.integration.jira.DefaultJiraService] communicateWithJira Problem communicating with JIRA: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
-- referer: http://localhost:8090/spaces/viewspace.action?key=HIPAA | url: /rest/highlighting/1.0/panel-items | traceId: a5047c1f57aa8ce6 | userName: alexander.korzun
2020-03-05 14:52:33,415 WARN [http-nio-8090-exec-2] [confluence.plugins.createjiracontent.CachedJiraResourcesManager] getRemoteJiraServer Could not detect issue creation supported or not in Jira server The server may be unreachable. Please ensure you are running JIRA 5.0 or higher.
-- referer: http://localhost:8090/spaces/viewspace.action?key=HIPAA | url: /rest/highlighting/1.0/panel-items | traceId: a5047c1f57aa8ce6 | userName: alexander.korzun
2020-03-05 14:52:42,514 WARN [Caesium-1-3] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:57:42,515 WARN [Caesium-1-2] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job LaasPerformanceLoggingJob#LaasPerformanceLoggingJob completed unsuccessfully with response JobRunnerResponse[runOutcome=ABORTED,message='LaaS performance logging is turned off']
2020-03-05 14:58:31,653 INFO [localhost-startStop-4] [atlassian.plugin.manager.DefaultPluginManager] shutdown Preparing to shut down the plugin system
2020-03-05 14:58:31,670 INFO [localhost-startStop-4] [atlassian.plugin.manager.DefaultPluginManager] shutdown Shutting down the plugin system
2020-03-05 14:58:34,953 WARN [Gemini Blueprint context shutdown thread1] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] destroy atlassian-instrumentation-jmx expose scheduler stopped.
2020-03-05 14:58:54,013 INFO [Gemini Blueprint context shutdown thread2] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] lambda$stop$9 Stopping Synchrony...
2020-03-05 14:58:54,056 INFO [Gemini Blueprint context shutdown thread2] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] lambda$stop$9 Stopping Synchrony...
2020-03-05 14:58:54,058 WARN [157:StdOutHandler [c:\Progra~1\Java\jre1.8.0_144\bin\java.exe]] [atlassian.utils.process.ExternalProcessImpl] maybeFinish Finishing c:\Progra~1\Java\jre1.8.0_144\bin\java.exe -classpath C:\Program Files\Atlassian\Confluence\temp\2.1.0-release-confluence_6.5-1a01ab2d.jar;C:\Program Files\Atlassian\Confluence\confluence\WEB-INF\lib\mssql-jdbc-6.3.0.jre8-preview.jar -Xss2048k -Xmx1g synchrony.core sql finished
java.lang.NoClassDefFoundError: com/atlassian/confluence/plugins/synchrony/bootstrap/DefaultSynchronyProcessManager$1
at com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyProcessManager$SynchronyProcessMonitor.onAfterFinished(DefaultSynchronyProcessManager.java:281)
at com.atlassian.utils.process.ExternalProcessImpl.notifyAfterFinished(ExternalProcessImpl.java:601)
at com.atlassian.utils.process.ExternalProcessImpl.wrapUpProcess(ExternalProcessImpl.java:669)
at com.atlassian.utils.process.ExternalProcessImpl.finish(ExternalProcessImpl.java:192)
at com.atlassian.utils.process.ExternalProcessImpl$AsynchronousOutputHandlerRunnable.maybeFinish(ExternalProcessImpl.java:730)
at com.atlassian.utils.process.ExternalProcessImpl$AsynchronousOutputHandlerRunnable.run(ExternalProcessImpl.java:724)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: Unable to load class 'com.atlassian.confluence.plugins.synchrony.bootstrap.DefaultSynchronyProcessManager$1' because the bundle wiring for com.atlassian.confluence.plugins.synchrony-interop is no longer valid.
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1494)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 9 more
2020-03-05 14:59:04,214 ERROR [FelixShutdown] [internal.util.concurrent.RunnableTimedExecution] execute Closing runnable for context NonValidatingOsgiBundleXmlApplicationContext(bundle=net.customware.confluence.plugin.vault, config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms; consider taking a snapshot and then shutdown the VM in case the thread still hangs
2020-03-05 14:59:14,220 ERROR [FelixShutdown] [internal.util.concurrent.RunnableTimedExecution] execute Closing runnable for context NonValidatingOsgiBundleXmlApplicationContext(bundle=net.customware.confluence.plugin.scaffolding, config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms; consider taking a snapshot and then shutdown the VM in case the thread still hangs
2020-03-05 14:59:24,226 ERROR [FelixShutdown] [internal.util.concurrent.RunnableTimedExecution] execute Closing runnable for context NonValidatingOsgiBundleXmlApplicationContext(bundle=com.servicerocket.confluence.plugin.servicerocket-utility-library, config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms; consider taking a snapshot and then shutdown the VM in case the thread still hangs
2020-03-05 14:59:24,226 WARN [Gemini Blueprint context shutdown thread1] [beans.factory.support.DisposableBeanAdapter] destroy Invocation of destroy method failed on bean with name 'defaultLicenseChecker': org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.upm.api.license.PluginLicenseEventRegistry)] unavailable
2020-03-05 14:59:34,229 WARN [Gemini Blueprint context shutdown thread2] [beans.factory.support.DisposableBeanAdapter] destroy Invocation of destroy method failed on bean with name 'licenseChecker': org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.upm.api.license.PluginLicenseEventRegistry)] unavailable
2020-03-05 14:59:34,417 ERROR [localhost-startStop-4] [hipchat.spacetoroom.service.SpaceToRoomNotificationService] hipChatNotifications Failed to obtain hipchat notifications for event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@5e0db0e9: service proxy has been destroyed
2020-03-05 14:59:34,417 WARN [localhost-startStop-4] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.plugin.event.events.PluginFrameworkShutdownEvent@5e0db0e9 took 62754 ms (warning threshold is 5000 ms)
2020-03-05 14:59:34,479 ERROR [localhost-startStop-4] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'contentPropertyService' threw an exception
com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.getServiceTracker(OsgiServiceRegistryImpl.java:75)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.invoke(OsgiServiceRegistryImpl.java:80)
at com.sun.proxy.$Proxy205.hashCode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.replaceNode(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.getServiceTracker(FelixOsgiContainerManager.java:490)
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.getServiceTracker(FelixOsgiContainerManager.java:482)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$1.create(OsgiServiceRegistryImpl.java:46)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$1.create(OsgiServiceRegistryImpl.java:43)
at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
... 27 more
2020-03-05 14:59:34,495 ERROR [localhost-startStop-4] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'boostingQueryFactory' threw an exception
com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.getServiceTracker(OsgiServiceRegistryImpl.java:75)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.invoke(OsgiServiceRegistryImpl.java:80)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1415)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.getServiceTracker(FelixOsgiContainerManager.java:490)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
... 27 more
2020-03-05 14:59:34,526 ERROR [localhost-startStop-4] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'outboundWhitelist' threw an exception
com.atlassian.util.concurrent.LazyReference$InitializationException: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:149)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.getServiceTracker(OsgiServiceRegistryImpl.java:75)
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.invoke(OsgiServiceRegistryImpl.java:80)
at com.sun.proxy.$Proxy191.hashCode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.replaceNode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.remove(Unknown Source)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1415)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Unable to create a tracker when osgi is not running
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.getServiceTracker(FelixOsgiContainerManager.java:490)
at com.atlassian.plugin.osgi.container.felix.FelixOsgiContainerManager.getServiceTracker(FelixOsgiContainerManager.java:482)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
... 27 more
2020-03-05 14:59:34,557 INFO [localhost-startStop-4] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2020-03-05 14:59:34,573 ERROR [localhost-startStop-4] [atlassian.confluence.setup.ConfluenceListableBeanFactory] destroyBean Destroy method on bean with name 'apiNavigationService' threw an exception
com.atlassian.confluence.osgi.OsgiNoServiceAvailableException: No service available for com.atlassian.confluence.rest.api.services.RestNavigationService
at com.atlassian.confluence.osgi.OsgiServiceRegistryImpl$ServiceTrackingProxy.invoke(OsgiServiceRegistryImpl.java:83)
at com.sun.proxy.$Proxy165.hashCode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.replaceNode(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.remove(Unknown Source)
at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.postProcessBeforeDestruction(PersistenceAnnotationBeanPostProcessor.java:374)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1426)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1415)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2020-03-05 14:59:34,589 INFO [localhost-startStop-4] [com.atlassian.confluence.lifecycle] shutdownCacheManager Shutting down EhCache cache manager
2020-03-05 14:59:34,604 INFO [localhost-startStop-4] [com.atlassian.confluence.lifecycle] contextDestroyed Stopping Confluence

[Ajay _view26_]

Please have a look at the thread  which talks of a similar issue. Kindly pos your full catalina.out and atlassian-confluence.log we should be able to help you figure out what is going on.

[gaurav.trivedi]

Hello Ajax, 

Thank you for your reply,

We have updated main post with new logs

Thanks & Regards

Gaurav Trivedi

[Ajay _view26_]

Seems like a known  issue . Can you please try out the workaround and provide an update.