Hello Confluence admins,
I have recently upgraded Confluence from 7.19.17 to 8.5.12.
This is a two node cluster and has been working just fine until the upgrade.
The cluster is connected to an SQL server and uses kerberos to authenticate.
Below the configuration used and modified to remove sensitive info.
The pool size has been 60 for the last few years and I have increased it to 80 in an attempt to stabilize the service
Configuration
confluence.cfg.xml
<property name="hibernate.c3p0.max_size">80</property>
<property name="hibernate.c3p0.validate">true</property>
<property name="hibernate.connection.autocommit">false</property>
<property name="hibernate.connection.driver_class">com.microsoft.sqlserver.jdbc.SQLServerDriver</property>
<property name="hibernate.connection.isolation">2</property>
<property name="hibernate.connection.password">REDACTED</property>
<property name="hibernate.connection.provider_class">com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider</property>
<property name="hibernate.connection.url">jdbc:sqlserver://HOST:PORT;databaseName=DATABASE;domain=DOMAINNAME;integratedSecurity=true;authenticationScheme=JavaKerberos</property>
<property name="hibernate.connection.username">REDACTED</property>
<property name="hibernate.database.lower_non_ascii_supported">true</property>
<property name="hibernate.dialect">com.atlassian.confluence.impl.hibernate.dialect.SQLServerDialect</property>
<property name="hibernate.hikari.idleTimeout">30000</property>
<property name="hibernate.hikari.maximumPoolSize">80</property>
<property name="hibernate.hikari.minimumIdle">20</property>
<property name="hibernate.hikari.registerMbeans">true</property>
server.xml
<Connector port="8080" connectionTimeout="20000" redirectPort="8443"
maxThreads="48" maxPostSize="16777216" minSpareThreads="10"
maxHttpHeaderSize="24578" enableLookups="false" acceptCount="10" URIEncoding="UTF-8"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https" secure="true" proxyName="confluence.domain.tld" proxyPort="443" />
Issue
Error
At least twice a day one of the two nodes will stop responding. Sometimes they both stop responding. The logs fill up with 'Unable to aquire JDBC Connection' due to no connections in the pool, due to No loginModule found. (See Exception Exanmple below)
Analysis
The DBA team are confident there are no issues and they do not see failed authentication attempts.
Restarting Confluence brings the node online and it starts accepting traffic, logs indicate normal operation
Observation
The bundled SQL Server driver is used and quite old (7.4) mssql-jdbc-7.4.1.jre8.jar
From the database-jdbc-drivers documentation
We recommend that you use the bundled Type 4 JDBC driver.
If you decide to use a later version, we may not be able to provide support for any problems you encounter.
The bundled jar is only for java 8 if it does hwat it says on the tin
Java
I do not find a bundled version of Java in the gunzip archive
| Confluence version | Supported Java vendors | Bundled Java vendor |
|---|
| 7.13.2 to 7.13.x, and 7.14.1 to latest | Oracle JDK/JRE
Adopt OpenJDK
Eclipse Temurin | Eclipse Temurin |
EDIT:
From supported-platforms the JRE is not expected to be in the gunzip file. It is only in the installer
Good to know:
- You don't need to install Java if you plan to use the installer to install Confluence, as a Temurin Java 17 JRE is bundled with Confluence.
END EDIT
Exception Example
2024-08-12 01:31:30,166 WARN [Caesium-1-4] [confluence.impl.search.IndexFlushScheduler] lambda$new$1 Failed to flush index queue LuceneContentIndexFlusher, retry in 10s
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Co
nnection
at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:600)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:31)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at jdk.internal.reflect.GeneratedMethodAccessor125.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
at com.sun.proxy.$Proxy107.getTransaction(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
at com.sun.proxy.$Proxy239.flushQueue(Unknown Source)
at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:122)
at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121)
at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83)
at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:464)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJob(CaesiumSchedulerService.java:431)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:409)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:516)
at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:502)
... 28 more
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.getConnection(HikariCPConnectionProvider.java:77)
at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.getConnection(DelegatingHikariConnectionProvider.java:48)
at jdk.internal.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:113)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
at com.sun.proxy.$Proxy743.getConnection(Unknown Source)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 31 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Kerberos Login failed: Integrated authentication failed. ClientConnectionId:bc429792-b23f-4c23-9af8-b98f1a769208 due to javax.security.auth.login.Logi
nException (No LoginModule found for com.sun.security.auth.module.Krb5LoginModule)
at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:100)
at com.microsoft.sqlserver.jdbc.KerbAuthentication.generateClientContext(KerbAuthentication.java:207)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4820)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3731)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.access$000(SQLServerConnection.java:94)
at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3675)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2488)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2142)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1993)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1164)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:760)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
Caused by: javax.security.auth.login.LoginException: No LoginModule found for com.sun.security.auth.module.Krb5LoginModule
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:731)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581)
at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthInit(KerbAuthentication.java:77)
... 23 more
