We've been running confluence in a VM (vmware) for around 6 month with no issue. But, suddenly it stops responding. From the server we found the confluence is not running. When we try to run it manually (in foreground), the process gets killed within few minutes after it starts. Hers the console log:
[root@confluence]# /opt/atlassian/confluence/bin/start-confluence.sh -fg
executing using dedicated user: confluence
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /opt/atlassian/confluence/jre//bin/java
2019-04-14 15:10:14,312 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
Using CATALINA_BASE: /opt/atlassian/confluence
Using CATALINA_HOME: /opt/atlassian/confluence
Using CATALINA_TMPDIR: /opt/atlassian/confluence/temp
Using JRE_HOME: /opt/atlassian/confluence/jre/
Using CLASSPATH: /opt/atlassian/confluence/bin/bootstrap.jar:/opt/atlassian/confluence/bin/tomcat-juli.jar
Using CATALINA_PID: /opt/atlassian/confluence/work/catalina.pid
14-Apr-2019 15:10:14.785 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2019 15:10:14.865 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '1' did not find a matching property.
14-Apr-2019 15:10:14.961 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2019 15:10:14.967 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2019 15:10:15.009 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2019 15:10:15.029 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
14-Apr-2019 15:10:15.311 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
14-Apr-2019 15:10:15.324 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
14-Apr-2019 15:10:15.331 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 654 ms
14-Apr-2019 15:10:15.338 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
14-Apr-2019 15:10:15.339 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.8
14-Apr-2019 15:10:16.992 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer@6ce3c1ee]
SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
2019-04-14 15:10:17,709 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.10.1 [build 7702 based on commit hash c32f84c806260a834c2e109e904a2d8ee4f8e175] - synchrony version 2.1.0-master-ed94c233
Killed
What can cause this issue and how do we fix this?
Hey @Kamrul Khan ,
What version of Confluence are you running?
The symptoms you described match an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen attackers infect servers with malware that tries to mine bitcoin. The malware runs under the confluence system user account, consumes all the CPU, and relaunches itself every few minutes if you kill the process.
If you're on a vulnerable version, the first step in fixing this is upgrading Confluence. The latest releases are:
Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.
Please let me know if you have more questions!
Daniel | Atlassian Support
Have you check the log files after you did start confluence (especially
/opt/atlassian/confluence/logs/catalina.out)
Cheers
Kurt
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Even I faced a similar issue on confluence 6.3.1
When tried to start the service it ends up showing similar errors, finally restarted the server and it worked as nothing happened.
What could be the possibilities for these errors?
15-Apr-2019 07:32:10.430 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.551 VARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.575 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a mat ching property.
15-Apr-2019 07:32:10.587 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
15-Apr-2019 07:32:10.647 VARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did n ot find a matching property.
15-Apr-2019 07:32:11.660 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
15-Apr-2019 07:32:11.772 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
15-Apr-2019 07:32:11.777 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8443"]
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.