Need help updating SSL Cert for JIRA Software and Service Desk

[Majid Mirpour]

Hello

I have been tasked with replacing the existing, expired SSL, with a new one, on our JIRA Service Desk and Software server as well as our JIRA Confluence server.  I was told by a member of our security team port 443 on the servers is showing that the SSL cert has expired.  I am not a JIRA admin at all, I am Linux system admin and I have zero idea of how to go about doing this without breaking the server.  The documentation provided by JIRA is confusing for someone with no experience with JIRA at all.   Any suggestions on how I can do this?

We are using: 

JIRA Service Desk 4.10.1

Jira Software 8.10.1

Jira Core 8.10.1

Atlassian Confluence 6.6.0

Thanks!

[Daniel Ebers]

Hi@Majid Mirpour

the setups I came across had a reverse proxy in front of Jira/Confluence - this is a common thing because terminating SSL connection at proxy levels is much easier than at JVM level.

That leads to the question: do you have a Apache or nginx based reverse proxy in front of your Atlassian products?

If so, usually (but without any guarantee for your setup we don't know) renewing a Cert is more of replacing a file, probably reloading the process (Apache or nginx) and you're all set.

Could you please describe your setup any more detailed, do you use a reverse proxy?

Tasked with questions like this I feel documentation is key - as soon this all is sorted out it might be a good idea to document where the configuration must be made so fellow admins of yours can find those information easily.

Cheers,
Daniel

[Brant Schroeder]

@Majid Mirpour 

Welcome to the community.  SSL can be fun because there are so many factors that can determine your success.  Do you happen to have a test instance that is setup the same as your production?   Is your production instance on a VM?

I would suggest starting with your test instance.  If you do not have one then I would suggest taking a snapshot of your VM so you have something to role back to.  (DB and Server)  

Really though you should review the guide you were talking about.  https://confluence.atlassian.com/adminjiraserver0810/running-jira-applications-over-ssl-or-https-1014674543.html

If you are running a proxy or have multiple instances on a single server (Confluence and Jira running on same server), then you could have additional issues.  

Since you are on Linux it should be easier since it is similar to adding SSL to any web application running on Linux by ensuring it is in the Java Keystore and properly referenced in jira's configuration.

[Majid Mirpour]

@Brant Schroeder 

Hello Brant!

Thank you for the reply! Yes we do have a staging instance of the server.  Jira Service Desk and Jira Software are running as two models on the same server that Jira Core is running on.   Confluence is running on a separate server.  If i was to follow the instructions in the documentation that you just provided me in your reply, will the existing SSL cert on the server be replaced with the new one?  Also are there any tips you can provide me that will help me with this task?

Thanks!

[Brant Schroeder]

@Majid Mirpour Great news that you have a staging server to test on.  Is the test instance a VM that you can take a snapshot of?   This will allow for easy rollback if something does not go right.  

Yes the documentation would take you through the steps to replace the SSL certificate.  As I stated before it would be good to know if you have a proxy setup because that changes things a bit.  Do you happen to know if you are running a proxy?  You can check the config to see if a proxy is running.

Apache - Look at the server.xml - https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html

NGINX - Look at the server.xml - https://confluence.atlassian.com/jirakb/configure-jira-server-to-run-behind-a-nginx-reverse-proxy-426115340.html

As far as tips go I would just make sure that you can recover easily.  This way if anything goes wrong you can start over.  Even if everything goes well it might be a good idea to do it a couple of times in stage to confirm steps before moving to production.

[Majid Mirpour]

@Brant Schroeder 

@Daniel Ebers 

Hello Brant and Daniel

Thank you for the replies. Yes we have an F5 which acts as proxy to our  VM thats running Jira. I am checking if we have a separate F5 which acts as the proxy for our Confluence server.  I will let you know what I find out.  and Yes we can take a snapshot of the JIRA VM. That is not a problem. Our DB for the JIRA server is running on a separate VM. Should I take a snapshot of the DB VM as well?

Thanks!

[Daniel Ebers]

It's never wrong to have one - if I got everything right your task is about replacing a SSL cert, though.
Given the assumption this is done on F5 level, in this particular case a database backup won't be needed specifically.

[Majid Mirpour]

@Daniel Ebers 

Basically we have one F5 which has different virtual hosts on it for the proxies with different IP addresses. They are logically seperated proxies.   The URL for the JIRA site, is hosted on the JIRA server, but when you browse to the site, you hit the F5 proxy.. when you browse to the site, the SSL cert shows as valid in the browser.  So I am thinking that the expired SSL cert is actually located on the JIRA VM itself and not the F5.  

[Brant Schroeder]

@Majid Mirpour You can view the details of the cert in the browser and find out if that is the one hosted on the F5.  I would agree with @Daniel Ebers  that the cert is most likely on the F5.  The security scan could have easily discovered the expired cert on the server depending on how it is scanning servers (Agent installed on the server).  If this is the case sounds like the F5 is delivering a valid certificate but the one on the server is expired.   

[Majid Mirpour]

@Brant Schroeder 

@Daniel Ebers 

Thank you for the reply.  The security scan was scanning the actual servers themselves and not the F5.  I know this because the F5 certificate is valid but the cert on each of the servers is expired.  That being said, what method should I use to update the certs on each server ?  

[Daniel Ebers]

That's hard to say from the outside, without knowing your specific deployment.
Normally it is replacing a very few files, reloading Apache/nginx are you're good to go.
Best would be maybe to check with your sys admin (or whoever setup the installation for you).

[Majid Mirpour]

@Daniel Ebers 

@Brant Schroeder 

I have asked like 4 people and no one knows how this is set up.   I am totally stuck.  

[Daniel Ebers]

This is usually a good point in time to renovate the setup. As soon as the operation team does not understand it - it shows something must gone wrong.

You could start explaining what is in use, maybe this rings a bell with some of us here.

In general I'd search for a installation acting as a reverse proxy, then it's configuration files and then specifically anything referring to a private key (the part of the SSL certificate as such).

But, however - without knowing your site it is guesswork at it's best.

[Majid Mirpour]

@Daniel Ebers 

Thank you for the reply.  That would be way too much work.  I was not part of the installation team for this product, in fact I wasnt even part of the company when it was installed.  My only task is to get the SSL updated.. thats it. and then I can close the ticket in our ticketing system..  so trying to renovate the system is not something I can pursue.   I can bring it up to the team, but I dont think they would really go that route.   I just want to know, if I follow the instructions in the document  that you provided to me earlier, will the SSL cert on the servers itself, (NOT the F5) be updated.  That is all.  

[Daniel Ebers]

Yes, the documents are valid - but you will need to find out where the certificate must be replaced. It could be in JVM or in a reverse proxy.

Brant already described how to determine if this is the case. This is the first step I'd do.

Apache - Look at the server.xml - https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html

NGINX - Look at the server.xml - https://confluence.atlassian.com/jirakb/configure-jira-server-to-run-behind-a-nginx-reverse-proxy-426115340.html

[Majid Mirpour]

@Daniel Ebers 

Thank you for the reply.  Honestly the only thing I ever do with JIRA, is log into the portal and create tickets or update tickets or close them.   So this is all very new to me.  I did do the following: I reached out to one of our network engineers and he informed me that he already updated the SSL certificate on the F5 for the Staging environment.  The F5 is a VIP server and its not really a "reverse proxy" per se as per what he told me.  Now as for where on the actual JIRA server, the SSL certificate resides on, that is a whole different thing.  When I log into the JIRA server itself, I do not see any directories for httpd nor nginx on the server.  I am wondering if the SSL configuration for this server is defined in the virtual host configuration on the staging F5 VIP device.   At my previous job, I had to set up an Apache based reverse proxy as an IPv6 to IPv4 proxy and if I recall correctly, I had to set up the SSL configuration for each virtual host that was being proxied, in the virtual host configuration file for the Apache based reverse proxy and NOT on each of the servers themselves.   Is there anyway I can check if the SSL certs are stored in a JVM? 

Thanks

[Daniel Ebers]

Sure, using keytool.

This is basically what Brant mentioned referring to the following link:

https://confluence.atlassian.com/adminjiraserver/running-jira-applications-over-ssl-or-https-938847764.html#RunningJiraapplicationsoverSSLorHTTPS-commandline

[Majid Mirpour]

@Daniel Ebers 

Thank you for the reply.  I took a look at the server.xml file and this is what I see for the keystore information:

I had to change the server name and password, for security purposes, before pasting to this post of mine:

<Connector relaxedPathChars="[]|" relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;" port="8080" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false"
maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" redirectPort="8443"
acceptCount="100" disableUploadTimeout="true" bindOnInit="false"/>

<Connector relaxedPathChars="[]|" relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;" port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxHttpHeaderSize="8192" SSLEnabled="true"
maxThreads="150" minSpareThreads="25"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS" useBodyEncodingForURI="true"
keyAlias="stage-jira-server" keystoreFile="/etc/pki/jira/stage-jira-server.ks" keystorePass="RaNd0Mp@$$w0rD"=" keystoreType="JKS"/>

Does this give any indication if the SSL cert is stored in a JVM or in the proxy?

Thanks

[Daniel Ebers]

Could be - at least somebody adapted the keystoreFile - /etc/pki/jira/stage-jira-server.ks (this is not default).

You could try adding your cert to there, according to https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html

[Majid Mirpour]

@Daniel Ebers 

The instructions on the site you linked me to, seems like they are for setting up an SSL connection between two different servers.  The server.xml config I posted to you is from our Jira Service Desk/Software server.  Would the instructions you linked me to, still apply to an SSL certificate for a single server?  I am focused on the Service Desk/Software server first and then I will worry about the Confluence one.

Thanks!

[Majid Mirpour]

@Daniel Ebers 

@Brant Schroeder 

I worked with our network engineer and took a look at the F5 reverse proxy configuration and I can confirm that the expired certificate does NOT reside on the proxy.  It is on the server itself.  That being said, will the instructions that you linked me to in this link, work with a cert that resides on the server itself?

https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html

The reason I ask this is because the website describes the problem that it is trying to solve as:  "When connecting two servers via HTTPS, the public SSL certificate from each server must be added to the other server's JVM truststore."

Whereas, all I am trying to do is update the certificate on the server itself, not trying to connect two servers together.  

Thanks

[Brant Schroeder]

@Majid Mirpour 

Per your configuration you will need to import the certificate into the keystore on the server so the information that @Daniel Ebers provided is correct.  Depending on how your F5 is configured you will most likely be required to have the F5 admin add the certificate the F5 keystore.  

[Majid Mirpour]

@Brant Schroeder 

Thank you for the reply.

The first line of the command line instructions has the following:

Fetch the certificate, replacing google.com with the FQDN of the server JIRA is attempting to connect to:

Which server is the certificate supposed to be fetched from? Can't I just generate the CSR on the JIRA server and then get the certificate from our CA and then copy it over to the JIRA server and then insert it into the keystore?

[Brant Schroeder]

Which server is the certificate supposed to be fetched from? Can't I just generate the CSR on the JIRA server and then get the certificate from our CA and then copy it over to the JIRA server and then insert it into the keystore? YES

You can actually follow the initial instructions that I believe you posted on how to setup SSL.  You will most likely have to share the certificate with 

https://confluence.atlassian.com/adminjiraserver0810/running-jira-applications-over-ssl-or-https-1014674543.html

[Majid Mirpour]

@Brant Schroeder 

Thank you for the reply. I tried to insert the certificate into the keystore, and unfortunately it did not work.  The certificate that I insert has the intermediate and the root certificate all in the same file.  Should I only insert the cert itself and not the chain and root certs along with it?  

This is getting even more confusing for me

[Majid Mirpour]

@Brant Schroeder 

@Daniel Ebers 

Here is a quick update of what I have done so far: I followed the instructions in the running jira applications over ssl or https document, and I recreated the keystore, as well as inserted the root and intermediate certificates into the keystore.  But when I try to insert the actual signed SSL certificate into the keystore, I get the following error:

keytool error: java.lang.Exception: Public keys in reply and keystore don't match

When I created the CSR, I used the private key that I generated use the openssl command. I then submitted the CSR to our CA and I downloaded the signed chain certificate.  I then separated the root, intermediate and signed certs from the main signed chain certificate, into 3 files. I then inserted each of those certificates into the newly generated keystore, but ONLY when I try to insert the actual signed key, I get the error that I posted above.  Please keep in mind that I am using the correct keystore password each time I run any keytool command on this server.  Do you have any idea what might be the cause of this error and how I should fix it?

oh and when I was creating the keystore, I got a question asking to enter a key  or press enter if the key is the same as the keystore password, I pressed enter at that moment, and the keystore was created. 

Thanks!

[Majid Mirpour]

@Brant Schroeder 

@Daniel Ebers 

Any suggestions on what I should do? 

Thanks

[Majid Mirpour]

@Brant Schroeder 

@Daniel Ebers 

I think I figured out how to do this.  I was able to install the certificate in the Java Keystore and I see when I browse to the website.  Now I Just have to ask Security to scan the server again to verify that the cert is there.

Here is what I had to do:

1) Generate the private key

2) Generate the CSR

3) Submit the CSR to the CA to get the certificate in *.p7b format

4) Convert the *.p7b format to a *.crt

6) Export a *.pfx file from the *.crt file (Make sure you provide an export password)

7) I then used the instructions on this post to import the *.pfx keystore into a new Java Keystore.

https://community.atlassian.com/t5/Confluence-questions/Update-Expired-SSL-Certificate-in-Confluence-Server/qaq-p/1333173

8) I changed the alias of the new keystore from 1 to my desired alias using the instructions provided on this site:

http://devnumbertwo.com/change-alias-keystore-using-keytool/

9) I restarted the Jira service using the stop-jira.sh and start-jira.sh scripts

10) Reloaded the website and verified that the new cert is there!

I am so happy! haha