Confluence crashes suddenly - "kernel too old"

[BTICINO]

Goodmorning all,

since a couuple of day my Confluence instance crashes suddenly few seconds after being restarted. The only error the server shows me in command line is:

FATAL kernel too old

After that, the JAVA process crashes and Confluence is not available anymore.

Catalina.out logs are not showing anything about this error. I'm not able to find this error anywhere at all.

If i try to restart Confluence, it run for a few minutes and then it crashes again. While Confluence is running I was able to check the healt status (from <my.weg.instance>/plugins/servlet/troubleshooting/view/) and everythig was fine.

I tried also to start Confluence in safe mode, disabling all addons, but it didn't work.

Environment details are:

Confluence version is: 6.11.1

Server OS is: Linux 2.6.32-573.el6.x86_64 #1 SMP Wed Jul 1 18:23:37 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

JAVA version is:  "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

Tomcat version is: 9.0.10

GNU libic version is: 2.12

Total free memory is: 6 GB (total 8 GB)

Swap free memory is: 2 GB (total 2 GB)

Filesystem free space is: 68% (13 GB)

Does anyone know something about this issue? Is there a way to solve it?

[Diego]

Hello there! So, Grigory pointed out that your issue looks like his. And he might be right.  With this in mind, here goes the same advisory:

Based on your version and symptoms, it sounds like your instance might be affected by an opportunistic attack against the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20). We've seen an infection going around that injects malware and the bitcoin miner it tries to run uses all the CPU available on the box. Initially the kerberods malware was being deployed as the payload, but other attacks might be trying to inject different payloads.

I'd recommend tackling things in this order:

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

Malicious processes

The top command will help you find processes (probably running under the confluence user account) that are consuming a large amount of CPU. If Confluence is currently stopped, you can probably plan on killing any processes running as the confluence user. note the process ID (pid) from the top output and then kill the process using kill -9 followed by the pid. Example:

sudo kill -9 12395

Clean up your crontab

Since most malware adds a cronjob that relaunches the malware every few minutes, you'll also need to check the crontab file and remove any suspicious-looking entries. For Ubuntu, this is stored in the /var/spool/cron/crontabs/ directory. Normally you should use the crontab command to edit the crontab, but for cleanup purposes we'll be inspecting the file for any pre-existing entries.

Using vim (or whichever text editor you're comfortable with), you'll open the file and remove suspicious-looking jobs.

sudo vim /var/spool/cron/crontabs/confluence

Confluence comes up on system startup through the SysV/systemd daemons, so we would expect the confluence user's crontab to not exist under normal circumstances. It's most likely the case that any entries in this file are malicious, but make sure you check them before deleting them entirely.

Upgrade Confluence

Once your CPU is under control and new malicious process aren't spawning, you need to upgrade Confluence to a version that isn't affected by the vulnerability. I'd recommend looking at one of these versions (latest releases as of this post):

• 6.13.4 (6.13 is an Enterprise release)
• 6.14.3
• 6.15.2

Use a malware scanner

Finally, you need to clean up any remaining traces of malware on your system. The LSD malware cleanup tool will be useful for removing the Kerberods malware. Other malware payloads might need different cleanup tools depending on which attack and payload were used. A good starting place for detecting other types of infections are the scanners linked here. Once a particular infection is identified, googling for "____ removal tool" is a good place to start if the scanner was unable to remove the malware automatically.

Taking in consideration your application version and symptoms, it is likely that you are affected by this vulnerability.

[BTICINO]

Hello Diego,

according to your suggestions, I checked my confluence user's processes, and the only one running was JAVA. Tha amount of CPU usage is very low and the memory usage is always around 40% (almost the maximum value allowed for conflunece). There is no crontab for confluence (as well as other users).

Following Confluence Security Advisory - 2019-03-20 - Mitigation steps, I disabled those addons

• WebDAV plugin
• Widget Connector

and now Confluence seems to run without problems.

Is it possible those addons are no longer compatible with my Conflunence version (6.11.1) or with my OS version (Linux 2.6.32-573.el6.x86_64)?

In this case, since I'm planning to upgrade Confluence version (from 6.11.1 to 6.13.4 as suggested), is it recommended to upgrade my system too (maybe from rhel 6 to rhel 7.5 or rhel 8)?

[Diego]

Hello again!

So, as far as compatibility goes, you should be able to run Connie 6.13.4 on the same OS you currently have.

The only unknown factor for me is if the database that you are currently using is compatible with Confluence 6.13.4. The only difference in database support between Connie 6.11.x and 6.13.x is PostgreSQL 9.3, which is not supported on 6.13.x.

I would suggest the following:

1. Backup everything that you have currently, <confluence-home> and <confluence-install> folders and your database
2. Perform a deeper malware check on your instance
3. Upgrade immediately to one of the fixed versions

Further, there is also a new CVE that was released yesterday. You can still upgrade to 6.13.4 because it is listed as one of the fixed versions. Here is the most current CVE:

Confluence Security Advisory - 2019-04-17 | Confluence - Path traversal vulnerability - CVE-2019-3398

It is unlikely that your issue was caused by incompatibilities in your system. Let us know!

[Grigory Salnikov]

Hi!

It looks just like my issue

How have you been able to find the "FATAL kernel too old" error? Where to look for it?

[BTICINO]

Hello,

Unfortunately it seems this error is not shown anywere. I didn't find it among linux system log files or confluence's log.
I'm able to see this error message only via command line..

No matter what I am writing or executing, the error message appears on the command line (and the java process is closed subsequently)

[Grigory Salnikov]

Ok, I don't see anything except "Killed"

There's another similar case:

Confluence Start Failure

[Grigory Salnikov]

All this is pretty strange, because the timing is perfect. My Confluence stopped working also a couple of days ago.

[BTICINO]

Strange coincidence indeed!

And apparently without any changes to configurations...

[Grigory Salnikov]

I used it on Friday but on Monday it was already dead. Something happened during the weekend.