Confluence had problems starting up - 6.15.2

[Christoph Vogler]

After upgrade to 6.15.2 my confluence instance is not starting any more.
See screenshot and catalina.out in the attachment.

I have checked possible solutions:
* https://confluence.atlassian.com/confkb/startup-check-confluence-won-t-start-as-it-cannot-connect-to-a-database-953127209.html

* https://confluence.atlassian.com/confkb/confluence-generates-confluence-is-vacant-error-on-install-779164449.html

I am able to access the database with the confluence user with a mysql client on the same machine, so mysql server seems to be up and running.

catalina.out:

05-Jun-2019 12:46:57.822 SEVERE [ajp-nio-127.0.0.1-10009-exec-9] org.apache.catalina.core.StandardHostValve.custom Exception Processing ErrorPage[errorCode=404, loc
ation=/fourohfour.action]
java.util.concurrent.CompletionException: com.atlassian.confluence.tenant.VacantException: Confluence is vacant, a call to tenanted [public abstract org.hibernate.
Session org.hibernate.SessionFactory.getCurrentSession() throws org.hibernate.HibernateException] is not allowed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
at java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:834)
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2155)
at java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:110)
at com.atlassian.confluence.impl.concurrency.CompletionStageUtils.foldResult(CompletionStageUtils.java:19)
at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
at com.atlassian.confluence.themes.DefaultThemeManager.getThemeSettings(DefaultThemeManager.java:71)
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalThemeKey(DefaultThemeManager.java:47)
at com.atlassian.confluence.themes.DefaultThemeManager.getGlobalTheme(DefaultThemeManager.java:40)
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:79)
at com.atlassian.confluence.setup.sitemesh.ConfluenceSpaceDecoratorMapper.getDecorator(ConfluenceSpaceDecoratorMapper.java:53)
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)
at com.atlassian.confluence.setup.sitemesh.PluginDecoratorMapper.getDecorator(PluginDecoratorMapper.java:32)
at com.opensymphony.module.sitemesh.mapper.AbstractDecoratorMapper.getDecorator(AbstractDecoratorMapper.java:45)
at com.opensymphony.module.sitemesh.mapper.InlineDecoratorMapper.getDecorator(InlineDecoratorMapper.java:41)
at com.atlassian.confluence.util.profiling.ConfluenceDecoratorSelector.selectDecorator(ConfluenceDecoratorSelector.java:64)
at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:83)
at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

[JP _AC Bielefeld Leader_]

Hi,

is this right at the beginning of the log? Are you using a JNDI connection managed by Tomcat or a connection configured in Confluence? What was the version you upgraded from? Which version has the MySQL server? Which JDBC driver version are you using?

Best

JP

[Christoph Vogler]

Hi Jan-Peter,

the connection is configured in the confluence.cfg.xml. The previous installed version was 5.10.2. MySQL server is at version 5.7.26. JDBC driver is 5.1.38.

The log file is full of the given error message. 

The first error message after a confluence restart is:

WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDe
tected Thread [ajp-nio-127.0.0.1-10009-exec-1] (id=[73]) has been active for [70,013] milliseconds (since [5/22/19 10:56 PM]) to serve the same request for [https:/
/confluence.chrizzly.at/] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are m
onitored by this Valve and may be stuck.
java.lang.Throwable
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:36)
at com.atlassian.hibernate.adapter.adapters.ConnectionProviderV5Adapter.getConnection(ConnectionProviderV5Adapter.java:35)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:514)

I have already increased the c3p0 timeout and pool size - same result.

Regards

Christoph

[Alexis Robert]

Hi @Christoph Vogler , 

I don't know what steps you've taken to diagnose the problem, but since this is an upgrade did you think about copying the mysql driver to the <jira_installation_dir>/lib folder ? 

For MySQL you can find it here: https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz

Let me know if this helps, 

--Alexis

[Christoph Vogler]

Hi Alexis,

thanks for the hint. The mysql driver is placed in the lib directory - an older version (5.1.38) but i think that does not matter.

Regards

Christoph