nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Confluence cluster setup

admin_n2rbz0

Infra

EFS (/share directory)
2 application aws nodes
application loadbalancer
postgresql (aws rds)

The above infra is setup in aws. Started with single node and then followed cluster setup document.

Node1 comes up and when trying to start node2 , node1 goes down with http 500 error.

If restarted service on node1 , node2 reports http 500 error.

node 2 (conf_access_log_

04.0.0.0 Safari/537.36
[17/Aug/2022:16:45:35 +0000] admin http-nio-8090-exec-1 10.0.254.143 GET /index.action HTTP/1.1 200 14175ms 9316 http://epo-ness-03b9c9.mvision-gpnes.local:8090/login.action?os_destination=%2Findex.action&permissionViolation=true Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

node1(conf_access_log)

[17/Aug/2022:17:17:16 +0000] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:16 +0000] - http-nio-8090-exec-4 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:36 +0000] - http-nio-8090-exec-3 10.0.188.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:36 +0000] - http-nio-8090-exec-5 10.0.235.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:56 +0000] - http-nio-8090-exec-9 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:56 +0000] - http-nio-8090-exec-6 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:18:16 +0000] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0

node 1 (catalina.out)

... 70 more
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.instance.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:321)
at com.hazelcast.instance.HazelcastInstanceProxy.getCluster(HazelcastInstanceProxy.java:219)
at jdk.internal.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy83.getCluster(Unknown Source)
at com.hazelcast.hibernate.HazelcastTimestamper.nextTimestamp(HazelcastTimestamper.java:38)
at com.hazelcast.hibernate.AbstractHazelcastCacheRegionFactory.nextTimestamp(AbstractHazelcastCacheRegionFactory.java:76)
at org.hibernate.internal.AbstractSharedSessionContract.<init>(AbstractSharedSessionContract.java:142)
at org.hibernate.internal.AbstractSessionImpl.<init>(AbstractSessionImpl.java:29)
at org.hibernate.internal.SessionImpl.<init>(SessionImpl.java:253)
at org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1229)
at org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:449)
at com.atlassian.confluence.impl.core.persistence.hibernate.SwitchableCachingSessionFactory.openSession(SwitchableCachingSessionFactory.java:117)
at jdk.internal.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy87.openSession(Unknown Source)
at com.atlassian.hibernate.adapter.bridge.factory.SessionFactoryV2orV5BridgeProxy.openSession(SessionFacto

I need active active cluster. But 2 nodes are not active at same time.

Accepted answers

All comments

Kenneth Epker

I have the exact same error. I'm running confluence through Kubernetes. Configured first node for clustering, no problems, shows up in the gui under confluence administration-> clustering as active. Increased replicas to 2, node 2 comes up and node 1 goes inactive with the errors described above. Now node 2 shows up in clustering and node 1 is no longer listed.

Yevhen_5guu4k

@Kenneth Epker did you enable clustering in values.yaml?

>Configured first node for clustering

How did you do that?

Kenneth Epker

Yes, I did enable clustering. I've copied the contents of local-home to shared-home and made sure the second node received a copy also, especially the confluence.cfg.xml. I've tried both Multicast and TCP/IP as the clustering type with no luck. I just came across a blog that mentions a Kubernetes connection type for clustering. Trying that now.

Kenneth Epker

I'm using the dso Ironbank helm chart to deploy my confluence DC product. I do not see any option to modify the confluence.cfg.xml file ahead of time to use "kubernetes" connection type. Do you have any suggestions?

Yevhen_5guu4k

I am not familiar with IronBank Helm Charts. I do recommend using the official Helm charts (you can still use IronBank Confluence image though)

Kenneth Epker

I'm sorry, didn't realize we changed our images to pull the docker.io/atlassian/confluence image via helm repo atlassian-github (v1.13.0). Looking through these charts though, I only see clustering connection types: Multicast, TCP/IP or AWS. None of these connection types will work in my kubernetes environment.

Yevhen_5guu4k

Correct, none of them will. All you need to do is to set confluence.clustering to true, Helm chart will do the rest. You said you migrated your local-home and I think that's the problem.

Can you please

1. Scale down to 0
2. set:

```

confluence:
additionalEnvironmentVariables:
- name: ATL_FORCE_CFG_UPDATE

value: "true"

```
This will overwrite cfg.xml in local home and set the right discovery type.
3. Run help upgrade
4. Make sure <property name="confluence.cluster.join.type">kubernetes</property> is in confluence.cfg.xml in local home
5. Scale to 2

Kenneth Epker

Great, trying it now.

Kenneth Epker

Not familiar with the step Run help upgrade. How do I do that?

Yevhen_5guu4k

I'm sorry - helm upgrade

Kenneth Epker

The node is hanging again. First it has a problem connecting to hazelcast now it just hangs on Could not get swap target filter. Below is the log:

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2

INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.

INFO:root:User is currently root. Will downgrade run user to confluence

INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']

executing as current user

If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

---------------------------------------------------------------------------

Using Java: /opt/java/openjdk/bin/java

2023-10-26 20:05:17,439 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required

---------------------------------------------------------------------------

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-10-26_20-05-17.log instead.

26-Oct-2023 20:05:18.122 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.130 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [proxyPort] to []

26-Oct-2023 20:05:18.138 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.142 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.202 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.278 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context/Valve] failed to set property [resolveHosts] to [false]

26-Oct-2023 20:05:18.280 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.621 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

26-Oct-2023 20:05:18.643 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [694] milliseconds

26-Oct-2023 20:05:18.679 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]

26-Oct-2023 20:05:18.679 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.73]

26-Oct-2023 20:05:21.093 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

26-Oct-2023 20:05:21.205 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath

26-Oct-2023 20:05:21.581 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'

2023-10-26 20:05:22,897 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.19.9 [build 8804 based on commit hash d47376fcfbd30b32890091b1ace5511f4b0beb75] - synchrony version 5.0.9

2023-10-26 20:05:26,707 WARN [Catalina-utility-1] [security.java8.serialfilter.DeserializationFilterConfigurator] configure Class sun.misc.ObjectInputFilter not available in classpath. Not setting JDK 8 DeserializationFilter.

java.lang.ClassNotFoundException: sun.misc.ObjectInputFilter

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1412)

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Class.java:315)

at com.atlassian.security.java8.serialfilter.DeserializationFilterConfigurator.configure(DeserializationFilterConfigurator.java:26)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at com.atlassian.confluence.setup.serial.ReflectiveFilterConfigurator.setReflectively(ReflectiveFilterConfigurator.java:11)

at com.atlassian.confluence.setup.serial.JDKFilterConfigurator.setJDKSerialFilter(JDKFilterConfigurator.java:23)

at com.atlassian.confluence.setup.DeserializationFilterConfigurator.configureDeserializationFilter(DeserializationFilterConfigurator.java:33)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:233)

at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.base/java.lang.Thread.run(Thread.java:829)

2023-10-26 20:05:26,885 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$5 Populating setup configuration if running with Cluster mode...

2023-10-26 20:05:27,692 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] configure Configuring Hazelcast with instanceName [confluence], join configuration com.atlassian.confluence.cluster.KubernetesClusterJoinConfig@0, network interfaces [10.42.2.52] and local port 5701

2023-10-26 20:05:27,693 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Starting the cluster.

2023-10-26 20:05:27,701 WARN [Catalina-utility-1] [com.hazelcast.instance.HazelcastInstanceFactory] log Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:

--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED

2023-10-26 20:05:30,381 INFO [hz.confluence.event-4] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] memberAdded [10.42.2.52]:5701 joined the cluster

2023-10-26 20:05:30,385 INFO [hz.confluence.event-4] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] logClusterMembers Cluster now has 1 members: [[10.42.2.52]:5701]

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/hazelcast-3.12.13.jar) to field sun.nio.ch.SelectorImpl.selectedKeys

WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

2023-10-26 20:05:30,540 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:32,043 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:34,296 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:37,676 WARN [Catalina-utility-1] [spi.discovery.integration.DiscoveryService] log [10.42.2.52]:5701 [GCCS] [3.12.13] Cannot fetch the current zone, ZONE_AWARE feature is disabled

2023-10-26 20:05:37,679 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:39,182 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:41,436 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:44,814 WARN [Catalina-utility-1] [spi.discovery.integration.DiscoveryService] log [10.42.2.52]:5701 [GCCS] [3.12.13] Cannot fetch name of the node, NODE_AWARE feature is disabled

2023-10-26 20:05:44,824 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:46,327 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:48,580 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:51,958 ERROR [Catalina-utility-1] [internal.cluster.impl.DiscoveryJoiner] log [10.42.2.52]:5701 [GCCS] [3.12.13] Failure in executing REST call

com.hazelcast.kubernetes.RestClientException: Failure in executing REST call

at com.hazelcast.kubernetes.RestClient.call(RestClient.java:114)

at com.hazelcast.kubernetes.RestClient.get(RestClient.java:79)

at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:503)

at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:497)

at com.hazelcast.kubernetes.RetryUtils.retry(RetryUtils.java:53)

at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:497)

at com.hazelcast.kubernetes.KubernetesClient.endpoints(KubernetesClient.java:80)

at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:70)

at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:135)

at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:71)

at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:70)

at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:59)

at com.hazelcast.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:151)

at com.hazelcast.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:111)

at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)

at com.hazelcast.instance.Node.join(Node.java:820)

at com.hazelcast.instance.Node.start(Node.java:455)

at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136)

at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)

at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:390)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:359)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:331)

at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:477)

at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:875)

at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)