Database pool Connection is not available

[Graham Twine]

Hello Confluence admins,

I have recently upgraded Confluence from 7.19.17 to 8.5.12.

This is a two node cluster and has been working just fine until the upgrade.

The cluster is connected to an SQL server and uses kerberos to authenticate.

Below the configuration used and modified to remove sensitive info.

The pool size has been 60 for the last few years and I have increased it to 80 in an attempt to stabilize the service

Configuration

confluence.cfg.xml

   <property name="hibernate.c3p0.max_size">80</property> 
   <property name="hibernate.c3p0.validate">true</property> 
   <property name="hibernate.connection.autocommit">false</property> 
   <property name="hibernate.connection.driver_class">com.microsoft.sqlserver.jdbc.SQLServerDriver</property> 
   <property name="hibernate.connection.isolation">2</property> 
   <property name="hibernate.connection.password">REDACTED</property> 
   <property name="hibernate.connection.provider_class">com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider</property> 
   <property name="hibernate.connection.url">jdbc:sqlserver://HOST:PORT;databaseName=DATABASE;domain=DOMAINNAME;integratedSecurity=true;authenticationScheme=JavaKerberos</property> 
   <property name="hibernate.connection.username">REDACTED</property> 
   <property name="hibernate.database.lower_non_ascii_supported">true</property> 
   <property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.SQLServerDialect</property> 
   <property name="hibernate.hikari.idleTimeout">30000</property> 
   <property name="hibernate.hikari.maximumPoolSize">80</property> 
   <property name="hibernate.hikari.minimumIdle">20</property> 
   <property name="hibernate.hikari.registerMbeans">true</property>

server.xml

        <Connector port="8080" connectionTimeout="20000" redirectPort="8443" 
                   maxThreads="48" maxPostSize="16777216" minSpareThreads="10" 
                   maxHttpHeaderSize="24578" enableLookups="false" acceptCount="10" URIEncoding="UTF-8" 
                   protocol="org.apache.coyote.http11.Http11NioProtocol" 
                   scheme="https" secure="true" proxyName="confluence.domain.tld" proxyPort="443" />
 

Issue

Error

At least twice a day one of the two nodes will stop responding. Sometimes they both stop responding. The logs fill up with 'Unable to aquire JDBC Connection'  due to no connections in the pool, due to No loginModule found. (See Exception Exanmple below)

Analysis

The DBA team are confident there are no issues and they do not see failed authentication attempts.

Restarting Confluence brings the node online and it starts accepting traffic, logs indicate normal operation

Observation

The bundled SQL Server driver is used and quite old (7.4) mssql-jdbc-7.4.1.jre8.jar

From the database-jdbc-drivers documentation

We recommend that you use the bundled Type 4 JDBC driver.

If you decide to use a later version, we may not be able to provide support for any problems you encounter.

The bundled jar is only for java 8 if it does hwat it says on the tin

Java

I do not find a bundled version of Java in the gunzip archive

Confluence version	Supported Java vendors	Bundled Java vendor
7.13.2 to 7.13.x, and 7.14.1 to latest	Oracle JDK/JRE Adopt OpenJDK Eclipse Temurin	Eclipse Temurin

EDIT:
From supported-platforms  the JRE is not expected to be in the gunzip file. It is only in the installer

Good to know:

• You don't need to install Java if you plan to use the installer to install Confluence, as a Temurin Java 17 JRE is bundled with Confluence.

END EDIT

Exception Example

2024-08-12 01:31:30,166 WARN [Caesium-1-4] [confluence.impl.search.IndexFlushScheduler] lambda$new$1 Failed to flush index queue LuceneContentIndexFlusher, retry in 10s 
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Co 
nnection 
       at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:600) 
       at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:31) 
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) 
       at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) 
       at jdk.internal.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) 
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
       at java.base/java.lang.reflect.Method.invoke(Method.java:566) 
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234) 
       at com.sun.proxy.$Proxy107.getTransaction(Unknown Source) 
       at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595) 
       at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382) 
       at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) 
       at com.sun.proxy.$Proxy239.flushQueue(Unknown Source) 
       at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:122) 
       at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121) 
       at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83) 
       at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134) 
       at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106) 
       at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:464) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:431) 
       at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:409) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60) 
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35) 
       at java.base/java.lang.Thread.run(Thread.java:829) 
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection 
       at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48) 
       at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37) 
       at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) 
       at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138) 
       at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:516) 
       at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:502)
        ... 28 more 
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms. 
       at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) 
       at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) 
       at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) 
       at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) 
       at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:77) 
       at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.getConnection(DelegatingHikariConnectionProvider.java:48) 
       at jdk.internal.reflect.GeneratedMethodAccessor237.invoke(Unknown Source) 
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
       at java.base/java.lang.reflect.Method.invoke(Method.java:566) 
       at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) 
       at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:113) 
       at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) 
       at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) 
       at com.sun.proxy.$Proxy743.getConnection(Unknown Source) 
       at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38) 
       at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108) 
       ... 31 more



Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Kerberos Login failed: Integrated authentication failed. ClientConnectionId:bc429792-b23f-4c23-9af8-b98f1a769208 due to javax.security.auth.login.Logi
nException (No LoginModule found for com.sun.security.auth.module.Krb5LoginModule) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:100) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.generateClientContext(KerbAuthentication.java:207) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4820) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3731) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$000(SQLServerConnection.java:94) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3675) 
       at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2488) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2142) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1993) 
       at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1164) 
       at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:760) 
       at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) 
       at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) 
       at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) 
       at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) 
       at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71) 
       at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726) 
       at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) 
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
       ... 1 more 
Caused by: javax.security.auth.login.LoginException: No LoginModule found for com.sun.security.auth.module.Krb5LoginModule 
       at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:731) 
       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672) 
       at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670) 
       at java.base/java.security.AccessController.doPrivileged(Native Method) 
       at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670) 
       at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581) 
       at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:77) 
       ... 23 more

[Graham Twine · Accepted solution]

The ms-sql driver shipped is now EOL and it is also for Java 8.
The error is due to the way Java 8 and Java 11 use ForkJoinPool.

:( If you are using MS SQL then Confluence is not really compatible with Java 11.