Confluence cluster setup

Infra

EFS (/share directory)
2 application aws nodes
application loadbalancer
postgresql (aws rds)

The above infra is setup in aws. Started with single node and then followed cluster setup document.

Node1 comes up and when trying to start node2 , node1 goes down with http 500 error.

If restarted service on node1 , node2 reports http 500 error.

node 2 (conf_access_log_

04.0.0.0 Safari/537.36
[17/Aug/2022:16:45:35 +0000] admin http-nio-8090-exec-1 10.0.254.143 GET /index.action HTTP/1.1 200 14175ms 9316 http://epo-ness-03b9c9.mvision-gpnes.local:8090/login.action?os_destination=%2Findex.action&permissionViolation=true Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

node1(conf_access_log)

[17/Aug/2022:17:17:16 +0000] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:16 +0000] - http-nio-8090-exec-4 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:36 +0000] - http-nio-8090-exec-3 10.0.188.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:36 +0000] - http-nio-8090-exec-5 10.0.235.214 GET /status HTTP/1.1 500 1ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:56 +0000] - http-nio-8090-exec-9 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:17:56 +0000] - http-nio-8090-exec-6 10.0.235.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0
[17/Aug/2022:17:18:16 +0000] - http-nio-8090-exec-8 10.0.188.214 GET /status HTTP/1.1 500 2ms 17 - ELB-HealthChecker/2.0

node 1 (catalina.out)

... 70 more
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.instance.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:321)
at com.hazelcast.instance.HazelcastInstanceProxy.getCluster(HazelcastInstanceProxy.java:219)
at jdk.internal.reflect.GeneratedMethodAccessor115.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy83.getCluster(Unknown Source)
at com.hazelcast.hibernate.HazelcastTimestamper.nextTimestamp(HazelcastTimestamper.java:38)
at com.hazelcast.hibernate.AbstractHazelcastCacheRegionFactory.nextTimestamp(AbstractHazelcastCacheRegionFactory.java:76)
at org.hibernate.internal.AbstractSharedSessionContract.<init>(AbstractSharedSessionContract.java:142)
at org.hibernate.internal.AbstractSessionImpl.<init>(AbstractSessionImpl.java:29)
at org.hibernate.internal.SessionImpl.<init>(SessionImpl.java:253)
at org.hibernate.internal.SessionFactoryImpl$SessionBuilderImpl.openSession(SessionFactoryImpl.java:1229)
at org.hibernate.internal.SessionFactoryImpl.openSession(SessionFactoryImpl.java:449)
at com.atlassian.confluence.impl.core.persistence.hibernate.SwitchableCachingSessionFactory.openSession(SwitchableCachingSessionFactory.java:117)
at jdk.internal.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy87.openSession(Unknown Source)
at com.atlassian.hibernate.adapter.bridge.factory.SessionFactoryV2orV5BridgeProxy.openSession(SessionFacto

I need active active cluster. But 2 nodes are not active at same time.

12 answers

0 votes

When doing initial setup, which is best, selecting Non-clustered or clustered? When I enable it during setup I only get the 3 options, Multicast, TCP, and AWS. I'm assuming I need to enable clustering mode in the GUI. This is the only way I get the <property name="confluence.cluster.join.type"></property> field in confluence.cfg.xml. Just having it enabled in my helm chart doesn't create the field in the cfg file. When I enabled cluster mode after setup and restart (delete k3s pod), the pod hangs and will not restart fully. I can exec into it and the confluence.cfg.xml is set back to "setupstart".

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

The node is hanging again. First it has a problem connecting to hazelcast now it just hangs on Could not get swap target filter. Below is the log:

INFO:root:Generating /opt/atlassian/confluence/conf/server.xml from template server.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/seraph-config.xml from template seraph-config.xml.j2

INFO:root:Generating /opt/atlassian/confluence/confluence/WEB-INF/classes/confluence-init.properties from template confluence-init.properties.j2

INFO:root:/var/atlassian/application-data/confluence/confluence.cfg.xml exists; skipping.

INFO:root:User is currently root. Will downgrade run user to confluence

INFO:root:Running Confluence with command '/opt/atlassian/confluence/bin/start-confluence.sh', arguments ['/opt/atlassian/confluence/bin/start-confluence.sh', '-fg']

executing as current user

If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

---------------------------------------------------------------------------

Using Java: /opt/java/openjdk/bin/java

2023-10-26 20:05:17,439 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required

---------------------------------------------------------------------------

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/logs/gc-2023-10-26_20-05-17.log instead.

26-Oct-2023 20:05:18.122 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.130 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [proxyPort] to []

26-Oct-2023 20:05:18.138 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.142 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.202 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.278 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context/Valve] failed to set property [resolveHosts] to [false]

26-Oct-2023 20:05:18.280 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host/Context] failed to set property [debug] to [0]

26-Oct-2023 20:05:18.621 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

26-Oct-2023 20:05:18.643 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [694] milliseconds

26-Oct-2023 20:05:18.679 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]

26-Oct-2023 20:05:18.679 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.73]

26-Oct-2023 20:05:21.093 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

26-Oct-2023 20:05:21.205 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log 1 Spring WebApplicationInitializers detected on classpath

26-Oct-2023 20:05:21.581 INFO [Catalina-utility-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring DispatcherServlet 'dispatcher'

2023-10-26 20:05:22,897 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.19.9 [build 8804 based on commit hash d47376fcfbd30b32890091b1ace5511f4b0beb75] - synchrony version 5.0.9

2023-10-26 20:05:26,707 WARN [Catalina-utility-1] [security.java8.serialfilter.DeserializationFilterConfigurator] configure Class sun.misc.ObjectInputFilter not available in classpath. Not setting JDK 8 DeserializationFilter.

java.lang.ClassNotFoundException: sun.misc.ObjectInputFilter

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1412)

at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1220)

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Class.java:315)

at com.atlassian.security.java8.serialfilter.DeserializationFilterConfigurator.configure(DeserializationFilterConfigurator.java:26)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at com.atlassian.confluence.setup.serial.ReflectiveFilterConfigurator.setReflectively(ReflectiveFilterConfigurator.java:11)

at com.atlassian.confluence.setup.serial.JDKFilterConfigurator.setJDKSerialFilter(JDKFilterConfigurator.java:23)

at com.atlassian.confluence.setup.DeserializationFilterConfigurator.configureDeserializationFilter(DeserializationFilterConfigurator.java:33)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:233)

at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)

at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4491)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4939)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)

at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.base/java.lang.Thread.run(Thread.java:829)

2023-10-26 20:05:26,885 INFO [Catalina-utility-1] [atlassian.confluence.cluster.DefaultClusterConfigurationHelper] lambda$populateExistingClusterSetupConfig$5 Populating setup configuration if running with Cluster mode...

2023-10-26 20:05:27,692 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] configure Configuring Hazelcast with instanceName [confluence], join configuration com.atlassian.confluence.cluster.KubernetesClusterJoinConfig@0, network interfaces [10.42.2.52] and local port 5701

2023-10-26 20:05:27,693 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Starting the cluster.

2023-10-26 20:05:27,701 WARN [Catalina-utility-1] [com.hazelcast.instance.HazelcastInstanceFactory] log Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:

--add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED

2023-10-26 20:05:30,381 INFO [hz.confluence.event-4] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] memberAdded [10.42.2.52]:5701 joined the cluster

2023-10-26 20:05:30,385 INFO [hz.confluence.event-4] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] logClusterMembers Cluster now has 1 members: [[10.42.2.52]:5701]

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/atlassian/confluence/confluence/WEB-INF/lib/hazelcast-3.12.13.jar) to field sun.nio.ch.SelectorImpl.selectedKeys

WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

2023-10-26 20:05:30,540 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:32,043 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:34,296 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:37,676 WARN [Catalina-utility-1] [spi.discovery.integration.DiscoveryService] log [10.42.2.52]:5701 [GCCS] [3.12.13] Cannot fetch the current zone, ZONE_AWARE feature is disabled

2023-10-26 20:05:37,679 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:39,182 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:41,436 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:44,814 WARN [Catalina-utility-1] [spi.discovery.integration.DiscoveryService] log [10.42.2.52]:5701 [GCCS] [3.12.13] Cannot fetch name of the node, NODE_AWARE feature is disabled

2023-10-26 20:05:44,824 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...

2023-10-26 20:05:46,327 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...

2023-10-26 20:05:48,580 WARN [Catalina-utility-1] [com.hazelcast.kubernetes.RetryUtils] log Couldn't discover Hazelcast members using Kubernetes API, [3] retrying in 3 seconds...

2023-10-26 20:05:51,958 ERROR [Catalina-utility-1] [internal.cluster.impl.DiscoveryJoiner] log [10.42.2.52]:5701 [GCCS] [3.12.13] Failure in executing REST call

com.hazelcast.kubernetes.RestClientException: Failure in executing REST call

at com.hazelcast.kubernetes.RestClient.call(RestClient.java:114)

at com.hazelcast.kubernetes.RestClient.get(RestClient.java:79)

at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:503)

at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:497)

at com.hazelcast.kubernetes.RetryUtils.retry(RetryUtils.java:53)

at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:497)

at com.hazelcast.kubernetes.KubernetesClient.endpoints(KubernetesClient.java:80)

at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:70)

at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:135)

at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:71)

at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:70)

at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:59)

at com.hazelcast.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:151)

at com.hazelcast.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:111)

at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)

at com.hazelcast.instance.Node.join(Node.java:820)

at com.hazelcast.instance.Node.start(Node.java:455)

at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136)

at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)

at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:390)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:359)

at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:331)

at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:477)

at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:875)

at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)

at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:236)