Hello,
Since 3rd of October the service is not working properly on node1 out of 4 of Confluence cluster.
Current errors from catalina.out logs are:
14-Oct-2024 02:55:06.079 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.296 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.308 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
14-Oct-2024 02:55:06.322 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [62851] milliseconds
Following these errors the service in systemctl appears running but in reality the node joins the cluster only at service start and then disconnects.
Looking back in the logs I see that the service was running just fine until September 19 after which there is a big interruption in the logs until 3rd of October when the issues started to appear:
19-Sep-2024 03:23:27.538 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8090"]
19-Sep-2024 03:23:27.542 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8090"]
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/latest/logs/gc-2024-10-03_20-50-40.log instead.
03-Oct-2024 20:50:41.394 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]
03-Oct-2024 20:50:41.928 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
03-Oct-2024 20:50:41.995 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]
On the 3rd of October we start to see errors:
2024-10-03 20:50:52,486 ERROR [Catalina-utility-1] [com.hazelcast.instance.AddressPicker] log [LOCAL] [Dell CPG Confluence] [3.12.14-atlassian-5] Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.
com.hazelcast.core.HazelcastException: Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.
03-Oct-2024 20:50:52.659 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8000] (base port [8000] and offset [0])
java.net.BindException: Address already in use (Bind failed)
2024-10-03 20:50:52,809 FATAL [Catalina-utility-2] [atlassian.spring.container.SpringContainerContext] getComponent Spring Application context has not been set
2024-10-03 21:05:38,479 WARN [Catalina-utility-2] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextDestroyed Failed to shut down plugin system during ServletContext shutdown: Spring Application context has not been set
03-Oct-2024 21:07:42.558 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
I have excluded node1 from load balancer so no traffic is forwarded towards it.
I have tried to gracefully power off and on the service using the confluence scripts (stop-confluence.sh & start-confluence.sh ), but no improvement.
A service restart using systemctl did not work also, neither a node restart.
I have checked file permissions and made sure that the service is running with confluence user.
I checked the database configuration from confluence.cfg.xml and everything is fine.
I compared the node with the other 3 which work just fine in regards of firewall rules and all is the same.
May I please have support on this issue?
Sorry I couldn't provide a support zip with the logs & config because the node does not stay connected long enough to download it.
Hi Robert,
Welcome to Atlassian community!
To resolve your issue, I recommend creating a ticket directly to Atlassian Support (Request assistance button on top), where you can describe the situation and include a log of the damaged Cluster in the attachment. Atlassian support will be able to help you more than "regular Atlassian users".
Pavel
You're right, thanks.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.