Atlassian apps not communicating after SSL install

[Al Twohill]

I've got nginx set up as a reverse proxy in front of Jira, Stash, and Bamboo. Each app is run from a separate nginx site on its own subdomain. 

Since adding SSL certificates to the nginx proxy the individual applications work fine but seem to be unable to communicate with each other. Any ideas?

[Al Twohill · Accepted solution]

Thanks to David's suggestions I've managed to find the solution

1. Download the certificate immediately above yours in the certificate chain. In my case this was the LetsEncrypt X1 Authority downloaded from https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem.txt 

2. Install it as follows. When prompted for a password it should be "changeit" : 

   /opt/atlassian/jira/jre/bin/keytool -import -alias letsencrypt-int -keystore /opt/atlassian/confluence/jre/lib/security/cacerts -file /root/letsencrypt-int.crt

3. Install for all other apps (replace the /opt/atlassian/confluence with the install path to the other apps)
4. Restart all apps
5. Link all apps

[Simon Karberg]

@Al Twohill

Do I need to do something special ?
I use Letsencrypt too, i have my fullchain cert that i can use for anything, should i link JIRA & Bitbucket Server to that file ?

I have NGINX with reverse proxy too.
Btw. i can't find cacerts for Bitbucket 4.5.2 

[Dave C]

cacerts exists in the Java trust store - this location will vary depending upon how you installed Atlassian apps and whether or not they came with a bundled JRE. Best thing is to check for the Java-home and cacerts will be located within that directory. This can be found in the System Info in JIRA and Support Tools > System Information in Bitbucket Server.

If you're using Letsencrypt, because it's a relatively new CA, you either need to the latest version of Java (as in contains the latest root CA certificaet) or put the root CA in the cacerts trust store.

[Dave C]

Anything in your application logs? One of the most common problems is there's a self-signed certificate, or the certificate doesn't exist in the Java trust store. We detail this in https://confluence.atlassian.com/display/KB/Unable+to+Connect+to+SSL+Services+due+to+PKIX+Path+Building+Failed

I'd suggest checking your site against https://globalsign.ssllabs.com and also ensure the best profile is used from https://mozilla.github.io/server-side-tls/ssl-config-generator/. Other problems might be the cipher / protocol is not supported by a particular version of Java that the applications are running on

[Al Twohill]

Well I get a grade A from ssllabs.com - https://globalsign.ssllabs.com/analyze.html?d=jira.twohill.nz&s=2600%3a3c00%3a0%3a0%3af03c%3a91ff%3afe18%3a2295 

It's possible that Java isn't up to the current spec or doesn't have the latest root certs  - the certs are generated by letsencrypt.org.. is there a local test I can do to check that they are valid? all my sites are on the same server. 

[Dave C]

Ahh I think that's going to be the issue. Lets encrypt uses a relatively new CA certificate, so the ca certs probably don't exist in older trust stores (ie older Java versions). According to http://stackoverflow.com/questions/34110426/does-java-support-lets-encrypt-certificates it looks like 8u66+ includes those certs. Easiest thing to do here would be to import the root lets encrypt ca certs into the trust stores as per that KB I linked earlier.

[Al Twohill]

OK. I've imported the root lets encrypt ca into all the various apps' keystores, but I still see errors such as "the application at URL 'https://wiki.twohill.nz' is not responding. Please confirm that you want to use this URL."

[Dave C]

There may have been failures during creation of the AppLinks due to the SSL certificates - I'd suggest trying to recreate them to see if this resolves the problem. If not, please try configuring them outside HTTPS, as per https://confluence.atlassian.com/display/KB/How+to+create+an+unproxied+application+link.

Also there may be errors / exceptions in the application logs to check and see what's going on.

[Al Twohill]

When you say application log, what exactly do you mean? I've been looking at catalina.out in the various apps, is there something more useful?

[Al Twohill]

I am still seeing things like 

Server at stash.twohill.nz/45.33.22.98:7999 presented unverified RSA key: 3f:61:77:0d:40:07:f6:38:8f:22:3f:dc:11:c5:02:9c - maybe I'm missing a step?

[Mike Friedrich]

... SHA256RSA in tomcat. For nginx check their documentation.

[Mike Friedrich]

Check that the algorithms used to sign is available in the Java version (or whichever part validates them) used.

It must be able to validate them.

Save to use should be