Could not open Hibernate Session for transaction; Hazelcast instance is not active!

[Lara Weber]

Hello,

We have Confluence Data Center with 2 nodes. The second node was not reachable for a few hours. I got the the error message:

ERROR [ConfluenceSpringScheduler:thread-9] [security.denormalisedpermissions.impl.DenormalisedChangeLogListenerBase] onCronEvent Unable to process records: Could not open Hibernate Session for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:598)
    at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:31)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
    at jdk.internal.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy107.getTransaction(Unknown Source)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
    at com.sun.proxy.$Proxy157.getSpaceServiceState(Unknown Source)
    at com.atlassian.confluence.security.denormalisedpermissions.impl.space.DenormalisedSpaceChangeLogListener.isServiceEnabled(DenormalisedSpaceChangeLogListener.java:73)
    at com.atlassian.confluence.security.denormalisedpermissions.impl.DenormalisedChangeLogListenerBase.onCronEvent(DenormalisedChangeLogListenerBase.java:95)
    at jdk.internal.reflect.GeneratedMethodAccessor121.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
    at com.hazelcast.instance.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:321)
    at com.hazelcast.instance.HazelcastInstanceProxy.getCluster(HazelcastInstanceProxy.java:219)
    at jdk.internal.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy117.getCluster(Unknown Source)
    at com.hazelcast.hibernate.HazelcastTimestamper.nextTimestamp(HazelcastTimestamper.java:38)
    at com.hazelcast.hibernate.HazelcastLocalCacheRegionFactory.nextTimestamp(HazelcastLocalCacheRegionFactory.java:77)
    at org.hibernate.cache.spi.AbstractCacheTransactionSynchronization.<init>(AbstractCacheTransactionSynchronization.java:19)
    at org.hibernate.cache.spi.StandardCacheTransactionSynchronization.<init>(StandardCacheTransactionSynchronization.java:14)
    at org.hibernate.cache.spi.AbstractRegionFactory.createTransactionContext(AbstractRegionFactory.java:145)
    at org.hibernate.internal.AbstractSharedSessionContract.<init>(AbstractSharedSessionContract.java:159)
    at org.hibernate.internal.AbstractSessionImpl.<init>(AbstractSessionImpl.java:29)
    at org.hibernate.internal.SessionImpl.<init>(SessionImpl.java:230)
    at org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1334)
    at org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:486)
    at com.atlassian.confluence.impl.core.persistence.hibernate.SwitchableCachingSessionFactory.openSession(SwitchableCachingSessionFactory.java:120)
    at jdk.internal.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy108.openSession(Unknown Source)
    at org.springframework.orm.hibernate5.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:479)
    ... 28 more

 What can I do to avoid the system error?

Thank you and Kind Regards

[Shreya Gupta]

Hi, so the fix we applied is - creating a clusterrole and clusterrolebinding for confluence and destinationrule setting the value of ttl to 0 for now and adding the httpcookieheader too.

Post implying the cr and crb, got the issue resolved.

[Automation Group]

Hi Shreya,

Great to know, you find the solution. Would be great if you elaborate more. 

Regards,

Ankit Gupta

[fivahij carspure]

@Shreya Gupta, hi

can you explain more detailed your fix, please?

[Automation Group]

Hi..!

Do we have any solution for it? if yes kindly suggest.

As per this article, we don't have any solution -

https://jira.atlassian.com/browse/CONFSERVER-64166

Regards,

Ankit Gupta

[Shreya Gupta]

Did you got the solution - i am also facing the same issue - for version 8.6.2

[IKP Reddy]

we also faced the same situation, As of now we confluence working on both nodes fine. 

Now suddenly we see the above issue in node2. after restart it up.

But not sure why it suddenly gave the error. we are in 8.5 version.

Any one has know the root cause here ?

[Gita Meskauskas]

Does anyone find couse of error? I get wthat error on version 8.5.4 too.

[Joel Erickson]

Also affecting ver. 8.5.9

[Kenneth Epker]

Experiencing the same error when attempting to add a node in confluence clustering using Kubernetes.

[Martin Podhovnik]

Hi!

If you run your DC cluster on Linux, I would have a look on the firewalld or selinux settings.

Just had the same issue when setting up another node - and reason was a forgotten port allowance.

BR Martin

[Gita Meskauskas]

Hi, We met the same situation too. Which version of Confluence you are running? We are on 7.19.14. Any ideas what's wrong?

[Gonchik Tsymzhitov]

Hi! I met with absolutely the same situation

[Gonchik Tsymzhitov]

It has been by updating Confluence