Huge SSL bug in apache: How to upgrade tomcat in confluence?

[samjones6]

Running Confluence 5.2.3 on a win2003x86 server.

A huge security hole has been discussed : http://arstechnica.com/security/2014/04/critical-crypto-bug-in-openssl-opens-two-thirds-of-the-web-to-eavesdropping/

I am rather certain that the tomcat bundled with confluence is vulnerable.

a) Where is a version of tomcat that is patched?

b) How to install the patch in a running Win2003x86 server?

thank you!

[Tiago Comasseto]

I belive your question is related to the Heartbleed bug and how it is affecting Atlassian applications. This has been confirmed, after investigations, to be an infrastructure issue. Our findings have been published to this blog that outlines the issue context, please take some time to review the details here:

http://blogs.atlassian.com/2014/04/openssl-cve-2014-0160-atlassian<br< a="">>
To find out whether you are affected or not, please check the version of OpenSSL that is running on your server. OpenSSL 1.0.1 or 1.0.2 releases may be affected. The entirety of OpenSSLs statement regarding this matter can be accessed after this link:

https://www.openssl.org/news/secadv_20140407.txt<br< a="">>
Cheers

[samjones6]

note, url above is bad. correct url is:

http://blogs.atlassian.com/2014/04/openssl-cve-2014-0160-atlassian/

[samjones6]

Wait a minute. That blog:

http://blogs.atlassian.com/2014/04/openssl-cve-2014-0160-atlassian/

Says to me: "The only reason you, the customer, have tomcat and openssl is because the confluence installer installed them. Yes, you now have an SSL cert on there. To patch up the openssl bug, that sits on your server because we gave it to you, you are on your own. Go figure it out."

What kind of support is that?

[Nic Brough -Adaptavist-]

Um, not quite. Yes, you have the Tomcat they've supplied, but it uses the OpenSSL that you installed on your server.

Just looking at the two installs I've got at home - one is vulnerable (well, was...), the other is not affected. Both installed from the same file from Atlassian, both SSL enabled, but on different operating systems.

[samjones6]

But I did not install openssl on this server... In all events, it is Atlassian's job to document how to evaluate whether a server needs help, and how to help it.

[Nic Brough -Adaptavist-]

They can't. It depends on what you've implemented on your server. They have no way of documenting every possible combination.

If you didn't install OpenSSL on the server, then either it's part of the operating system distribution (you need to check the version), you're not even using SSL (so you're fine), or you're using an SSL implementation that isn't based on OpenSSL (so you're fine).

How to fix a server is easy though, and on the advisory - upgrade OpenSSL to a fixed version, recreate all potentially affected certificates and change all your passwords.

Have a read of http://heartbleed.com/

https://www.schneier.com/blog/archives/2014/04/heartbleed.htmlfor a bit more of a human description

and try http://filippo.io/Heartbleed/as a scanner for your site(s)

[Joe Costello]

I agree with Sam on this and I am having a similar issue. I upgraded my Openssl to the newest version and I am still seeing the server as vulnerable. Thinking the issue is with Tomcat somehow but still not sure. Please post if you know what other steps, besides upgrading openssl, need to be completed to fix confluence on RHEL or CentOS. Thanks.

[Nic Brough -Adaptavist-]

Mmm, the problem is that Sam is basically wrong about the responsibility for documentation. The flaw is not in Atlassian's software, it's in supporting software that they rely on (and a metric shedload of other companies). There are tens of thousands, if not hundreds of thousands of software organisations affected and you expect all of them to document a fix to someone else's software? Atlassian, like the others, have said "move to a non-vulnerable version of SSL for your systems". . There's not a lot more they can do.

Anyway, that aside...

My vulnerable system was running on a distribution based on CentOS. Upgrading to the "latest" OpenSSL is not just a case of "yum upgrade", because the latest version in their repositories is still a vulnerable version.

You should check exactly which version you are running. Log in and:

nic@villa:~/ openssl

OpenSSL> version

OpenSSL 1.0.1g 7 Apr 2014

OpenSSL> exit

Note that mine now says 1.0.1g - the version in the repository was 1.0.1e. The bug is fixed in g, but I had to compile it myself for this distro. I'd expect RHEL and CentOS to catch up quite quickly, but you need to check...

[samjones6]

>> Responsibility for documentation

Ummmmm, the only reason tomcat exists in my entire rack is because Confluence put it there. It is installed by the Confluence installer.

We can argue all day about who is responsible. I know who is responsible: The installer that put it there.

[Joe Costello]

Here is exactly what I ran:

# wget http://www.openssl.org/source/openssl-1.0.1g.tar.gz

# tar –xf openssl-1.0.1g.tar.gz

# cd openssl-1.0.1g.tar.gz

# make clean

# ./config shared –-prefix=/usr –-openssldir=/usr/local/openssl

# make && make test

# make install

# openssl version

The correct version is showing as installed, however the confluence site is still vulnerable. I rebooted the server and I also tried a yum reinstall httpd but that didn't fix it either. Which was why I was thinking it could be a tomcat issue. Does this sound similar to what you did? Did you do anything else I could be missing?

Just looking to get patched, thanks for the help.

[Joe Costello]

BTW, I am using the python script from this website to check:

https://gist.github.com/takeshixx/10107280

[Nic Brough -Adaptavist-]

I also had to replace the openssl in /usr/bin on my system. But before I did this, http://filippo.io/Heartbleed/told me I failed, and afterwards, it worked.

I built from source almost exactly as you have - I left off the shared/prefix stuff, just doing ./config but I really don't expect that to be the problem for you. In fact, I suspect that's why I had to manually change the /usr/bin executable!

However... have you also regenerated all your certificates?

I'm using a self-signed certificate for my systems, and after installing the new version of OpenSSL, I deleted all the old ones and created new ones.

Unfortunately, I didn't actually check what happens if you use the old certificates with the new OpenSSL, I only checked after I'd upgraded OpenSSL, whacked in new certs and restarted everything.

[Nic Brough -Adaptavist-]

@Sam,

>> >> Responsibility for documentation

>>Ummmmm, the only reason tomcat exists in my entire rack is because Confluence put it there. It is installed by the Confluence installer.

>>We can argue all day about who is responsible. I know who is responsible: The installer that put it there.

I'm really sorry, but that is still wrong. The last thing we need during a major security problem is inaccurate information and pointing at the wrong things.

The Atlassian installers have installed Tomcat (amongst other things).

Tomcat uses OpenSSL when you configure it to use https.

Tomcat itself is not vulnerable. It only becomes vulnerable when it uses a version of OpenSSL that has the bug.

YOU installed OpenSSL. Probably as part of the operating system to be fair, but it's certainly not in the in Atlassian installers.

As I already said, I had a Confluence and Jira installation that was reported as vulnerable when tested. I upgraded OpenSSL on the server, restarted it, recreated all my certificates, and it was reported as not vulnerable thereafter. No other software has been touched. Exactly the same Confluence and Jira. Exactly the same Tomcats. Exactly the same Apache. The only change is that I've killed off the OpenSSL provided by my OS and replaced it with a later version.

So, as it's clearly not an Atlassian bug, and not a Tomcat bug, and is an OpenSSL bug, could you explain why it's Atlassian's responsibility to document it? Surely that would be like expecting IBM to document why Oracle's Java doesn't run their software very quickly on Microsoft Windows?

[Joe Costello]

I am using a Geotrust certificate on my server, and no I did not regenerate my certificate. I was thinking on doing that after this but you seem to think this could be the cause? Figured it was possible that someone stole the key BUT changing the OpenSSL version was the real fix and maybe some components of Tomcat/apache are still hung up on using the old version somehow? I upgraded other servers and they fixed pretty easily, my confluence server was the only one with this issue. It is one of the only apache/tomcat servers that I have with a trusted root CA certificate so maybe that is the issue.

I didn't setup the certs on this server, do you know of any docs on regenerating certificate requests for confluence? Or is this all done outside of confluence through tomcat?

Thanks for the help. I'm not blaming anyone, just looking to get it fixed..

[Tiago Comasseto]

To everyone,

Just a litlle clarification about the vulnerability and the tomcat shipped with Atlassian products. This is the explanation from our security engineer:

While the Tomcat we ship does include native libraries, if you followed our instructions for setting up SSL (https://confluence.atlassian.com/display/JIRA/Running+JIRA+over+SSL+or+HTTPS, same for other products), you end up using Java's own implementation of SSL, which is not vulnerable.

In other words, if in conf/server.xml you see something along the lines of

<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"

maxHttpHeaderSize="8192" SSLEnabled="true"

maxThreads="150" minSpareThreads="25"

enableLookups="false" disableUploadTimeout="true"

acceptCount="100" scheme="https" secure="true"

clientAuth="false" sslProtocol="TLS" useBodyEncodingForURI="true"/>

then Java SSL is used.

If for some reason you have (note the SSLCertificateFile and SSLCertificateKeyFile directives)

<Connector port="8443"

maxThreads="200"

scheme="https"

secure="true"

SSLEnabled="true"

SSLCertificateFile="${user.home}/certificate.pem"

SSLCertificateKeyFile="${user.home}/key.pem"

clientAuth="optional"

SSLProtocol="TLSv1"/>

then you are likely to be using vulnerable libraries. You can replace this configuration and switch to Java SSL by following instructions on the link above.

Cheers

[samjones6]

Nice write up! Good thing that is in the big Atlassian post on heartbleed. Would have answered a lot of questions if a proper write up had been presented by Atlassian to begin with.