Trojan/Virus in Hipchat Server 2.4?

[Neal Culiner]

As of around July 4th'ish Hipchat server would go down as the log directory was at 100%. I purged it, then 2 days later it happened again. Then today I was notified from a user on my mail server that mail wasn't working. I investigated and hipchat had tens of thousands of emails about the UK lottery winner blah blah. I shut down hipchat and it's still offline. Anyone else have this issue? How can I resolve it? I don't know of any anti-virus or ways of cleaning hipchat such as clamav. I'm running within Virtual Box on Windows Server which has symantec anti-virus scanning all servers. 

Thanks

[Neal Culiner · Accepted solution]

I ended up installing a new instance of HC Server and exported from the bad and imported to the new instance. All is fine now, no activity like before with the emails or log files. Had this been a compromised email account the issue would have remained in the new instance. Somehow malware got into HC 2.4.0. 

[Nic Brough -Adaptavist-]

<sigh> Wrong.

No one else has reported anything resembling this to Atlassian.  Therefore what you meant to say is that you or an attacker managed to install malware on it.

[Neal Culiner]

Nope, not what I meant to say, I said what I said. HC 2.4.0 is vulnerable to malware. We don't know what people say to Atlassian. I think those that saw what they did in mine were a bit surprised. Typical unix mentality that it's not vulnerable, why there isn't even Clam AV installed in the instance is odd.  All moot, HC is dead, hello Slack. 

[Nic Brough -Adaptavist-]

Nope, still wrong.

> We don't know what people say to Atlassian.

I asked.  Not only has no-one else reported this, if it had got malware in it, they would have pulled it completely and announced it.

So, the only possibility is that you or an attacker (via an insecurity in your systems) managed to install the malware on your system.

[Nic Brough -Adaptavist-]

It might be the case.  But that is extremely unlikely.  There are no reports that I can find with any version of Hipchat that are not down to an account being compromised.  If a virus or trojan somehow got into an Atlassian distribution, they would have removed if from download and told everyone not to use it.  They have not.

By far the most likely explanation is that one or more of your accounts have been compromised.

Go have a look at the account(s) that are causing this spam.  You are going to find that they have been hacked.

[Nic Brough -Adaptavist-]

Ah, I see.  I apologise, I misunderstood.

The way Hipchat is usually installed means that it's usually almost impossible to inject code into it, such as a virus or hack.

I suspect what has happened is one or more of your users has been compromised, and there are now 'bots logging into your Hipchat as that user and dumping spam into a room.

[Neal Culiner]

Today I did a hipchat upgrade --force-upgrade to hopefully replace the install with a fresh copy. With the IP address no longer whitelisted at least one of these two actions have mitigated the situation. Hopefully Atlassian didn't have a virus/trojan in the 2.4 install set.

[Nic Brough -Adaptavist-]

Again, there is no virus or trojan in it. 

The problem is that you have compromised accounts.

[Neal Culiner]

I don't believe it's compromised account as the IP address of the sender is the Hipchat Server IP address.  While the issue hasn't reoccurred, this may be due to the removing of the open gateway for this IP address. I'm still having issues with the log file folder reaching 100% and shutting me down. Working with Atlassian to see if there is anything they see. I still believe it's a virus/trojan causing it.

[Nic Brough -Adaptavist-]

That's a compromised account.   If it were a problem with hipchat, removing the open gateway would not fix it.

[Neal Culiner]

The emails started again yesterday so it's not an open gateway on the mail server which was removed. I again believe it's malware as I shut down the hipchat instance and the mails completely stop, restart the instance and the mails send again. I have since restored a backup to HC server 2.2.9 then reapplied the 2.4 update. We lost all conversations since then but so be it, we need to move on.

[Nic Brough -Adaptavist-]

If it is malware, I think the question becomes how you managed to install the malware into Hipchat.

[Neal Culiner]

Nothing but routine updates of HipChat. HipChat 2.2.9 to 2.4 would not install properly, the Chef package was corrupt so I followed Atlassian's guidance to download/install that and one KB article about passwords mismatching again per Atlassian. I think somewhere in there was malware OR it was a security issue known by Atlassian which drove the 2.4 update. I think all of the Tomcat clients are vulnerable right now and I'm waiting on Atlassian's updates across the board in their other products.

[Neal Culiner]

What may be the case is Atlassian shipped 2.4 with a virus/trojan. I unfortunately had the IP address whitelisted which may be how I learned of this. I have since removed the whitelist so SMTP AUTH is required. It doesn't mean everyone doesn't have the virus/trojan in HC Server 2.4.  I've seen this once in the past a software vendor unintentionally shipped a software update with a virus.

[Neal Culiner]

No, it's outgoing email sent from hipchat. We have anti spam at the firewall and mail server. 

[Nic Brough -Adaptavist-]

It does not sound like hipchat is the problem, it sounds like your email addresses have been hit with thousands of spam email which hipchat tries to process and fills up the disk logging it.

I think you need some anti-spam software on your email server, something to drop the incoming spam before hipchat looks at it.