Confluence node is not joining the cluster

Robert Tone
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
October 14, 2024

 

 

Hello,

 

Since 3rd of October the service is not working properly on node1 out of 4 of Confluence cluster.

Current errors from catalina.out logs are:

 

14-Oct-2024 02:55:06.079 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.296 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.308 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]

14-Oct-2024 02:55:06.322 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [62851] milliseconds

 

 

Following these errors the service in systemctl appears running but in reality the node joins the cluster only at service start and then disconnects.

Looking back in the logs I see that the service was running just fine until September 19 after which there is a big interruption in the logs until 3rd of October when the issues started to appear:

 

 

19-Sep-2024 03:23:27.538 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8090"]

19-Sep-2024 03:23:27.542 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8090"]

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/latest/logs/gc-2024-10-03_20-50-40.log instead.

03-Oct-2024 20:50:41.394 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

03-Oct-2024 20:50:41.928 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

03-Oct-2024 20:50:41.995 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]

 

On the 3rd of October we start to see errors:

 

2024-10-03 20:50:52,486 ERROR [Catalina-utility-1] [com.hazelcast.instance.AddressPicker] log [LOCAL] [Dell CPG Confluence] [3.12.14-atlassian-5] Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.

com.hazelcast.core.HazelcastException: Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.

03-Oct-2024 20:50:52.659 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8000] (base port [8000] and offset [0])

       java.net.BindException: Address already in use (Bind failed)

2024-10-03 20:50:52,809 FATAL [Catalina-utility-2] [atlassian.spring.container.SpringContainerContext] getComponent Spring Application context has not been set

2024-10-03 21:05:38,479 WARN [Catalina-utility-2] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextDestroyed Failed to shut down plugin system during ServletContext shutdown: Spring Application context has not been set

03-Oct-2024 21:07:42.558 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]

       org.apache.catalina.LifecycleException: Protocol handler initialization failed

 

 

I have excluded node1 from load balancer so no traffic is forwarded towards it.

I have tried to gracefully power off and on the service using the confluence scripts (stop-confluence.sh & start-confluence.sh ), but no improvement.

A service restart using systemctl did not work also, neither a node restart.

I have checked file permissions and made sure that the service is running with confluence user.

I checked the database configuration from confluence.cfg.xml and everything is fine.

I compared the node with the other 3 which work just fine in regards of firewall rules and all is the same. 

 

May I please have support on this issue?

 

Sorry I couldn't provide a support zip with the logs & config because the node does not stay connected long enough to download it.Screenshot 2024-10-14 105700.png

1 answer

1 accepted

1 vote
Answer accepted
Pavel Junek
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
October 15, 2024

Hi Robert,

Welcome to Atlassian community!

To resolve your issue, I recommend creating a ticket directly to Atlassian Support (Request assistance button on top), where you can describe the situation and include a log of the damaged Cluster in the attachment. Atlassian support will be able to help you more than "regular Atlassian users".

Pavel

Robert Tone
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
October 15, 2024

You're right, thanks.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events