High CPU Usage - isolating process

[IT KnowledgeSystemsManagement]

We recently experienced an hour long period of extremely high (90%+) CPU usage on our 16 core server which abide after about an hour. During this time no users reported any issues accessing Confluence and monitoring information from Wily indicated that response times were normal. For all intensive purposes the system was functional and within normal tolerances. However we would like to isolate what caused the CPU spike and were wondering if there is anyway to isolate which process would have caused this.

Regards,

-Christopher

[Simon Leclercq]

i have the same problem as i well

[Steve Petroskey]

Hi,

We are suddenly seeing the same issue.  Running top, the "khugepageds" process owned by confluence is pegged, reporting 198-200% CPU, load average of 2.3, and User CPU over 30%.

This is running on an AWS EC2, which I doubled, only to see it run fine for less than an hour and then max out again.  (Which kicks in AWS throttling, so it would probably consume all the CPU if it could.)

Before I doubled the CPU, the application wouldn't respond.  After that, it works with a slight delay, but is still maxed out.  Throwing more CPU at it will soon become unacceptable.

[David White]

@Steve Petroskey, in your situation "khugepageds" is almost certainly a rogue cryptocurrency miner, which someone has installed on your system via the WebDAV and Widget vulnerability

Kill the khugepageds processes, delete the rogue crontab entry for "confluence" that is running curl/wget processes, delete /tmp/kerberods (which has probably opened reverse SSH tunnels on your system), and reboot.  Then patch your system.

[Patrick du Boucher]

Hello,

I don`t have an answer, but we have exactly the same problem.

Twice in the last three weeks, the CPU went from 10% to about 30% for a couple of hours, and then spiked to 100% and never came down.

During this time, There were a lot of connections to the DB that were all idle, and usage in the application seemed unaffacted. Even collarborative editing was behaving correctly.

rebooting brought the CPU back to normal, but I don`t have a resolution, or explanation.

thanks

[DKB Service GmbH]

hi, we are facing absolutly the same issue since last week. do you have a resolution

[Fusion Helpdesk]

Hi,

Confluence Cpu: 13.7% and back to normal after rebooted server.

Thanks

[Fusion Helpdesk]

Hi Christopher,

We have submitted atlassian support ticket for this (https://getsupport.atlassian.com/servicedesk/customer/portal/14/CSP-202831) and monitoring very closely.

Thanks

[Joe Clark]

Hi Christopher,

Do you know if the spike is coming from Confluence, or some other process on your system?

For Confluence, the best thing to do is, when the spike starts happening begin taking a bunch of thread dumps using the built-in admin tool spaced 5-10 seconds apart and then submit them to https://support.atlassian.com for analysis.

If the spike is happening outside of Confluence, you'll need to resort to other ways that are dependent on your operating system. For Windows, you could use the built in performance monitoring tools to plot CPU usage by process over time and then see which process is spiking.

For *nix systems, you can use the top command.