Confluence crashed and now crashes when any Java process runs

Question

Confluence crashed for no reason after running fine for 6+ months on a T3 Medium EC2 instance. We can start it up, but then crashes the Java process when it processes the first request.

AMI ID: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20180912 (ami-00035f41c82244dab)

No obvious Confluence error logs.

Both Confluence instances Crashed on Sunday (two different servers) with the same problem.

Erwin Blomeyer · Answer

We installed 6.15.2 once we confirmed that the CPU profile had changed significantly on both Servers.

Just a bit disappointed that there was not a simpler process to patch and upgrade proactively from the console when there is such a massive threat / problem.

Daniel Eads · Answer

Hi Erwin,

What version of Confluence do you have installed currently? When you run top on the server, does it look like the CPU is running at 100% ?

The symptoms you described match an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen attackers infect servers with malware that tries to mine bitcoin. The malware runs under the confluence system user account, consumes all the CPU, and relaunches itself every few minutes if you kill the process.

If you're on a vulnerable version, the first step in fixing this is upgrading Confluence. The latest releases are:

* 6.6.13 (6.6 is an Enterprise release)
* 6.12.4
* 6.13.4 (6.13 is an Enterprise release)
* 6.14.3
* 6.15.2

Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!
Daniel | Atlassian Support