No access to Conlfuence Webpage, 100% CPU usage from "dbused"

[DK]

Hello together,

I have the problem that I can't connect to my Confluence instance, which I host on a virtual machine. If I look into htop I see that the command "dbused" from the user "confluence" is using 100% CPU and a lot of RAM. I'm not able to find anything useful searching the web.

If I restart confluence or even the server this behaviour will start again after a few minutes. If I kill this process it will start again after a short time.

Does anybody know what to do about that or under what keyword I can search for a solution?

I'm sorry if I'm missing something basic. We are a small research institute and we don't have "real" IT admins who are hosting our servers, so I have to do it with only intermediate linux/server knowledge...

Best regards

Dennis

[Alex Rangeo · Accepted solution]

Sounds like a miner of some type. Here is some of the steps I've taken, along with upgrading Confluence to the latest LTS.

Delete and Block access to a flagged file from Symantec -

rm -rf /tmp/dbused

chmod -R 000 /tmp/.pwn

Remove the cronjob that reloads the miner. It is running under the user that runs confluence.

crontab -e

You will see a job running every minute that is clearly attempting to connect and run rouge bash commands

Then looked up the host that the cron job communicated with, and added firewall rules to block the traffic. 

reboot

Upgrade to the latest LTS

[Scot Weir]

Thank you @Alex Rangeo , followed the above steps and Confluence has returned! Will be updating to LTS at the earliest opportunity.

[pavel shvetsov]

i was infected as well; you can read the "/tmp/xms" source script here 

http://bash.givemexyz.in/xms 

and find out that its better to check all these files and folders

/etc/cron.d/root /etc/cron.d/apache /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.hourly/oanacroner1 /etc/init.d/down 

 in my case i found crontab job inside `/var/spool/cron/crontabs/confluence2` file like Alex said

[wirat_leenavonganan]

I also found `/var/spool/cron/crontabs/confluence`

[Trung Vu]

The same isssue wwith Alex, it's come from everywhere. Is there any idea ?

[DK]

Thank you very much! It was indeed a miner. I think I could get rid of it.

Best regards

Dennis

[Trung Vu]

Hello @TUHH-DKA , i followed Alex steps, but it's still not fixed yet, can you provide some step.

The coinminer is automatic generate in /root/c3pool

[moemneimne]

Could this be the source https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html?

[Kamil Beer]

Hey Alex, thanks. I did both of these, yet to no result. The cron file (and the result process) regenerates every minute.

[Hanson Atlassian]

I had the same issue where 'dbused' process consumed massive CPU except mine wasn't launched via crontab, this was injected into my 'confluence' user's .bash_profile:

cp -f -r -- /tmp/.pwn/bprofr /tmp/dbused 2>/dev/null && /tmp/dbused -c  >/dev/null 2>&1 && rm -rf -- /tmp/dbused 2>/dev/null

everytime 'confluence' user logged in the 'dbused' process launched then removed itself from filesystem

[m_lang]

It seems that we had to start Confluence after applying the patch, provided by atlassian. 

Our  steps:

- Shut down confluence
- Apply Bugfix patch
- removed all cronjobs, located in /var/spool/cron/crontabs and /var/spool/cron/atjobs
- removed suspicious files in /tmp 
- killed dbused process, via kill -9 [pid]
- Start confluence

After confluence started the cronjob didn't regenerate under /var/spool/cron/crontabs. The dbused process was gone and no other suspicious  files in /tmp.

[Daniel Stiven Ballen Garcia]

De los archivos del /tmp cuales reconocen como maliciosos?

[m_lang]

I compared to another confluence instance, and the only folder there was: hsperfdata_confluence/

All other files owned by the confluence user were suspicious.

Without warranty, of course.

[Kamil Beer Vodafone]

Yeah, the confluence ownership is a giveaway, as the confluence user wouldn't have much of a reason to create the files there.

[Robert Wen_ReleaseTEAM_]

This sounds a lot like your Confluence server has a virus.

[Scot Weir]

Ran into this issue as well today for the first time.

On Confluence restart attempt, logs reveal:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f6500000, 32505856, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 32505856 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid23285.log

 free -m command shows memory dwindling slowly until approx 40MB available, then "resets" and begins the drain again.

We are running Confluence and Jira on AWS instances and truly do not want to move up to next size of instance type just to accommodate. I have tried making the recommended changes in the /tmp/hs_err_pid23285.log file with no success.

I have opened a support ticket with Atlassian in the meantime

[Michael O'Brien]

I have this issue as well starting at midnight on 1 Sept 2021 - bumping up my instance from 4g - but I was running fine on 70% ram utilization in the past

actually it is kworker (crypto) - I left 0.0.0.0/0 open - locking down the security group firewall

[Gustavo Jimenez]

i have the same problem on differents machine with confluence on AWS, all these machine has a CPU consume 100% o more

[Michael O'Brien]

20210901 crypto miner critical CVE - the patch worked for 2 weeks - restarted at 20210916:1800U - in the end upgrade from 6 to 7 LTS
https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html

[James Waithe]

Hi Michael

Are you saying that you applied the "atlassian workaround", but still got infected?

Thanks

Curious James

[AlexThunder]

maybe leftover cron or startup command?

[James Waithe]

Thanks

[Michael O'Brien]

Yes, I formally applied the workaround script in the CVE notice - totally fine for 2 weeks (before that in 1-6h the miner would restart).  To be honest I was extremely busy with multiple contracts and right when I was to refer to one of my wiki pages - the site went down on the 1st.  normally I would triage and mitigate - the workaround script worked well for 2 weeks.  Good thing it has restarted my side project to have a static generated site using S3/lambda/api-gateway serverless.  I also have the Atlassian confluence as a service but I like my own server/templates better.  Anyway I purchased every license I could a couple months ago before the standalone license was deprecated.  I used one of those licenses to keep my server going another year and updated from 6 to 7 - so far after a week I am good.

so yes, the workaround is not sufficient - it did not fully mitigate (either left the script or allowed for reinfection) - you MUST upgrade to 7

[AlexThunder]

It patches the vulnerability in confluence so you won't get new malware, but can't really solve the malware for you.. our confluence is still fine 24 days later. They could have included more details on what kind of malware has used the breach and where to look, but that already goes into normal server management.. which, if you are using the on-premise version, should know how to handle yourself.

https://tolisec.com/multi-vector-minertsunami-botnet-with-ssh-lateral-movement/
this has helped get some pointers on where to look

[Trung Vu]

The new coin miner type that i faced for this vulnerable: 

/tmp/kdevtmpfsi 

/tmp/syna -> it's remove all your iptables rules /etc/hosts

So please upgrade your Conflunece ASAP if you can.

[AlexThunder]

https://tolisec.com/multi-vector-minertsunami-botnet-with-ssh-lateral-movement/
this is also a good read about this malware. Will also apply the patch later. Seems like removing the crons and files don't help for long. It was fine for 1-2 hours, then i got another xmm monero crypto miner thing running with 100%. good thing they all get created in /tmp

[AlexThunder]

oh yes. this patch https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html did it for me.

[Kamil Beer]

Hey everyone,

after some time struggling, it seems that, at least for some time, I have managed to shut the miner, or whatever it was, down, and get Confluence running. What I did I did in a very short time window.

1) I removed the cron entry for confluence, which was located in /var/spool/cron

2) I killed all the suspicious processes, like the dbused

3) I applied Atlassian's patch.

From then, dbuserd didn't start, nor did the cronfile generate again.

Finally, I removed all the suspicious files from /tmp.

I'm now going to apply the LTE version.

[Daniel Stiven Ballen Garcia]

https://confluence.atlassian.com/doc/confluence-security-advisory-2021-08-25-1077906215.html

[Daniel Stiven Ballen Garcia]

@Kamil Beer de los archivos que encontro en  el /tmp cuales fueron sospechosos o que prefijo tenian?

Gracias 

[Kamil Beer Vodafone]

@Daniel Stiven Ballen Garcia, check how old they are, perhaps using "ls -lah". Most of the suspicious files were made on September 1st or 2nd (or whenever did this problem occur to you).

[Kamil Beer]

Same problem here. I delete the /var/spool/cron/confluence file, but it regenerates shortly and starts the dbused process again.

[Trung Vu]

try reboot the server, then quickly remove it

/var/spool/cron/confluence

/tmp/dbused

/tmp/.pwn

check the location as above comment: 

etc/cron.d/root /etc/cron.d/apache /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.hourly/oanacroner1 /etc/init.d/down 

upgrade the server to LTS version will help resolve it.

[Kamil Beer Vodafone]

@Trung Vu, yeah. The cronfile, for me, generated in /var/spool/cron, with the confoluence user. Anyway, it seems I resolved it using the workaround, but will update to LTS soon anyway.