Failed to join a node to the Confluence data center

Question

Hi everyone,

I'm trying to add a node to my Confluence data center instance and it's not working.

It shows me the following error screen every time I start the second node:

I've configured the second node with the steps that it puts in the Atlassian documentation, but the same thins happens constantly.

Both nodes see each other and have the correct ports open for hazelcast.

I show you a log trace:

2022-03-01 14:43:19,289 ERROR [Catalina-utility-2] [com.hazelcast.instance.Node] log [X.X.X.X]:5801 [node1] [3.12.11] Could not join cluster. Shutting down now!
2022-03-01 14:43:19,304 WARN [Catalina-utility-2] [com.hazelcast.instance.Node] log [X.X.X.X]:5801 [node1] [3.12.11] Terminating forcefully...
2022-03-01 14:43:19,373 WARN [Catalina-utility-2] [com.hazelcast.util.PhoneHome] log [X.X.X.X]:5801 [node1] [3.12.11] Could not schedule phone home task! Most probably Hazelcast failed to start.
2022-03-01 14:43:19,383 ERROR [Catalina-utility-2] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):
Node failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:138)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:384)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:353)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:325)
at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:427)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:815)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:69)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:193)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:145)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:63)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4766)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
2022-03-01 14:43:19,400 WARN [Catalina-utility-2] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextInitialized Not starting full plugin system due to upgrade
2022-03-01 14:43:19,404 [Catalina-utility-2]
[Filter: profiling] defaulting to off [autostart=false]

2022-03-01 15:03:06,505 ERROR [http-nio-8090-exec-1] [confluence.setup.johnson.WebAppContextPathCheck] check Bootstrap did not initialize.
2022-03-01 15:03:07,111 ERROR [http-nio-8090-exec-2] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/data | traceId: 002ea5e546a3cafc
2022-03-01 15:03:07,428 ERROR [http-nio-8090-exec-3] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/fav-confluence.ico | traceId: 004aef73ef48139b
2022-03-01 15:03:23,335 ERROR [http-nio-8090-exec-4] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/data | traceId: 001ad67dd2d34887
2022-03-01 15:03:39,335 ERROR [http-nio-8090-exec-6] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/data | traceId: 00243042413d077f
2022-03-01 15:03:55,332 ERROR [http-nio-8090-exec-8] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/data | traceId: 00c8eb6277120fd3
2022-03-01 15:04:11,341 ERROR [http-nio-8090-exec-10] [atlassian.confluence.util.AbstractBootstrapHotSwappingFilter] doFilter Could not get swap target filter
-- referer: https://X.X.X.X/ | url: /johnson/data | traceId: 00c8d1ed86866447

I appreciate your help in advance.

Mohamed EM · Answer

Hi Dave, sorry for the late reply.

Apparently there was an unknown external resource that was blocking the connectivity between the nodes.

Not having full visibility into the infrastructure, we were also unaware of this communication.

Thank you very much for your interest and response.

Dave Theodore [Coyote Creek Consulting] · Answer

Two things that we frequently see are that the host-based firewall does not allow the TCP traffic through.  This requires a modification to the host-based firewall. Used the example on the page I linked above, you would need to allow inbound TCP:8088 from all nodes in the cluster on each node.  Also check SElinux if you are running on linux.  You should see the blocks in your system logs. I would recommend using tcpping or some other utility that allows you to ping using TCP vs ICMP in order to validate the ports are open and that TCP traffic is flowing.

Additionally, you need to configure all nodes for unicast and adjust the configuration file on each node so that each node knows about all others.  You will have odd behavior if some nodes don't know about others. I hope that helps.

Mohamed EM · Answer

Hi Dave,

The servers are hosted on prem, the cluster configuration being used is TCP_IP.