Confluence server keeps crashing

Question

We've been running confluence in a VM (vmware) for around 6 month with no issue. But, suddenly it stops responding. From the server we found the confluence is not running. When we try to run it manually (in foreground), the process gets killed within few minutes after it starts. Hers the console log:

[root@confluence]# /opt/atlassian/confluence/bin/start-confluence.sh -fg

executing using dedicated user: confluence

If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide

Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out

---------------------------------------------------------------------------

Using Java: /opt/atlassian/confluence/jre//bin/java

2019-04-14 15:10:14,312 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required

---------------------------------------------------------------------------

Using CATALINA_BASE:   /opt/atlassian/confluence

Using CATALINA_HOME:   /opt/atlassian/confluence

Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp

Using JRE_HOME:        /opt/atlassian/confluence/jre/

Using CLASSPATH:       /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar

Using CATALINA_PID:    /opt/atlassian/confluence/work/catalina.pid

14-Apr-2019 15:10:14.785 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.

14-Apr-2019 15:10:14.865 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '1' did not find a matching property.

14-Apr-2019 15:10:14.961 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.

14-Apr-2019 15:10:14.967 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.

14-Apr-2019 15:10:15.009 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.

14-Apr-2019 15:10:15.029 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.

14-Apr-2019 15:10:15.311 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

14-Apr-2019 15:10:15.324 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read

14-Apr-2019 15:10:15.331 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 654 ms

14-Apr-2019 15:10:15.338 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]

14-Apr-2019 15:10:15.339 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.8

14-Apr-2019 15:10:16.992 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@6ce3c1ee]

SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are

SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.

SLF4J: See also http://www.slf4j.org/codes.html#replay

2019-04-14 15:10:17,709 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.10.1 [build 7702 based on commit hash c32f84c806260a834c2e109e904a2d8ee4f8e175] - synchrony version 2.1.0-master-ed94c233

Killed

What can cause this issue and how do we fix this?

Daniel Eads · Answer

Hey @Kamrul Khan ,

What version of Confluence are you running?

The symptoms you described match an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen attackers infect servers with malware that tries to mine bitcoin. The malware runs under the confluence system user account, consumes all the CPU, and relaunches itself every few minutes if you kill the process.

If you're on a vulnerable version, the first step in fixing this is upgrading Confluence. The latest releases are:

* 6.6.13 (6.6 is an Enterprise release)
* 6.12.4
* 6.13.4 (6.13 is an Enterprise release)
* 6.14.3
* 6.15.2

Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!
Daniel | Atlassian Support

AMAR_q4xpja · Answer

Even I faced a similar issue on confluence 6.3.1

When tried to start the service it ends up showing similar errors, finally restarted the server and it worked as nothing happened.

What could be the possibilities for these errors?

15-Apr-2019 07:32:10.430 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.551 VARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.575 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a mat ching property.
15-Apr-2019 07:32:10.587 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.647 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did n ot find a matching property.
15-Apr-2019 07:32:11.660 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
15-Apr-2019 07:32:11.772 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
15-Apr-2019 07:32:11.777 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8443"]

Kurt Klinner · Answer

@Kamrul Khan

Have you check the log files after you did start confluence (especially

/opt/atlassian/confluence/logs/catalina.out)

Cheers

Kurt