JIRA gadget rendering problem with client certificate requirement on Apache reverse proxy

[Jimmy Liberato]

I have a JIRA setup that uses an Apache reverse proxy (AJP protocol) front end with SSL server certificate. The reverse proxy just means that the SSL connection ends at Apache and Apache then forwards it to a special AJP connector in JIRA that listens on JIRA's port on localhost. This has worked just fine for 10 years.

My agency now requires that users must use client certificates in the form of a Common Access Card (CAC) as used by the government and military to see the server in addition to the normal SSL connection. No real problem there, either, since Apache has simple configuration file entries to require incoming connections to first present a client certificate signed by our agency.

That all works smoothly because the client certificate requirement is just to gain access to the server login screen to which they must login with their regular login credentials. I don't have to grok the users' credentials from the certificate or anything. It just needs to be valid and correctly signed.

After requiring client certificates there was an initial glitch with Application Links (Confluence is on the same physical server on a separate tomcat server). Application Links have always worked fine over SSL because the server SSL certificate was explicitly in the Java truststore. But the client certificate requirement munged Application Links communication via the Base URL so I just created a separate localhost-only http connector for both JIRA and Confluence to communicate with each other and that also works fine after configuring the remote application URL to be, for example, http://localhost:8090/confluence to connect to Confluence instead of the normal Base URL. That is an acceptable workaround.

The problem is with JIRA's change in how gadgets get rendered from 7.1 on. The client certificate requirement manifests the JIRA gadget title bug wherein titles appear like this: _MSG_gadget.activity, _MSG_gadget.filter, _MSG_gadget.project.title

See Health Check: JIRA Base URL  and How to fix gadget titles showing as __MSG_gadget

There is no immediate workaround that I can find because JIRA is hard-coded to use the Base URL (its external address) for accessing itself as explained in the link. I can't redirect it to localhost as with the Application Links to bypass Apache and its client certificate requirement. If I turn off the client certificate requirement in Apache then things work fine and the gadgets render correctly. So, basically Apache is rejecting JIRA's connection to itself because it presents no client certificate.

I am wondering if anyone has any suggestions on what I might look at to work around the title rendering problem. It is not show-stopping serious from a functional standpoint but it does look pretty shabby. I am going to look at iptables to see if there is a way to take any packets coming from the server to the server and redirect them to localhost instead. I haven't found any examples out there of that exactly so I will need to study up on iptables.

[Manse Wolken · Accepted solution]

If compute power is available: 

1. Put the Apache on its own Server

2. Put an apache on the Jira server, listening only on localhost network

3. Add your jira host to your /etc/hosts file (assuming you are on linux)

The new apache server (1) serves for the outside world and does the neat certificate access control. The other (2) serves only for jira (and wiki) local connections. This one does not require certificates (BUT has SSL enabled).

You might also be able to put those two apaches on one server, listening on different interfaces. 

[Jimmy Liberato]

Thanks.  I suspected it would be complicated like that.  One thing, though, wouldn't the JIRA requirement for using the Base URL (the external address) for connecting to itself still be an issue?  I can't tell JIRA to use the local web server for the Gadget rendering problem  like I can with Application Links.  JIRA will automatically use the external address on the external web server and therefore fail because of the darn client certificate requirement I have to impose.  JIRA either needs to offer up a client certificate somehow or I need to figure out how to redirect the traffic.  I don't think the first is possible and the second will require me to master IP Tables to find out if it is possible.

[Manse Wolken]

Thats more a trick of DNS and Hosts file.

Assuming you run Jira on Linux, make sure that the host running Jira uses the Hosts file before it uses DNS for name resolution. 

 In your hosts file, 127.0.0.1 stands for your jira.your.domain

DNS resolves to the correct external of jira.your.domain.

One Apache server has an interface binding to 127.0.01 port 443 and serves an ssl connection. Without the requirement of certificate checking.

The other Apache server has an Interface binding to your external IP. And this one enforces certificate checking.

So you basicly have an Apache Server, just for jira accessing ist own components.

The other one takes the real traffic.

If you want to use your Linux distributions tools, you might want to use apache and nginx.

The iptables way might work, but remember to configure your tomcat to serve SSL. otherwise it won't work either.

[Jimmy Liberato]

Ah, OK, I think I see now and that makes sense.  Thanks much for the clarification.  Since I've used nginx on another machine I think I will try that combo technique.

[Jimmy Liberato]

Manse, while playing around with some of your suggestions I discovered that in my particular case all I needed to do was to alias the FQDN (the JIRA basenane) to localhost 127.0.0.1 in the hosts file as you has mentioned.  Restarted networking and restarted Apache.  The solution almost seems too easy now! I think it worked so easily because I had already created the localhost connectors for the Application Links workaround which JIRA is now also using for the gadget rendering.   Anyway, thanks again for that key element in your suggestion. 

Edit : Ack!  It was just a cached page from when the client cert requirement was momentarily turned off.  I still have the gadget rendering problem.  No wonder it seemed too easy!   Oh, well...

[Manse Wolken]

Well at least you Jira now uses 127.0.0.1, so you might ant to consider the nginx // apache approach with listening on different interfaces.

Or, try the iptables approach (connections to 127.0.0.1:443 dnat 127.0.0.1:8443) 

Don't forget to configure ssl on tomcat and proxy_host // proxy_port options in server.xml.

[Jimmy Liberato]

Since Atlassian said they weren't going to fix the JIRA gadget title rendering bug and none of their proposed workarounds were relevant to the use of client certificates in a reverse proxy environment I decided I needed to attack this again. I basically used Manse's suggestions above. I created another virtual server in Apache to do the localhost part (after aliasing the FQDN part of the Base URL to 127.0.0.1 in the hosts file as above). I just copy and pasted the entire <VirtualHost> block under the original and made a couple modifications between them.

So, the original Apache settings looked like this:

<VirtualHost *:443>

...all of the SSL, the certificates and AJP proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

I needed to restrict the VirtualHost to just the external IP address rather than all addresses (*) and create another VirtualHost to handle forwarding from the host file (which is checked and acted on before external DNS gets a chance to send any internal requests to the external IP address). It now looks like this:

<VirtualHost [External IP Address goes here]:443>

...all of the SSL, the certificates and AJP reverse proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

<VirtualHost 127.0.01:443>

...all of the SSL, the certificates and AJP reverse proxy configuration entries in the original but including:

SSLVerifyClient none

</VirtualHost>

Connections originating from the local system (JIRA talking to itself) now bypass the SSLVerifyClient requirement and the gadget title rendering bug does not manifest itself.

Note that this is only a solution for those using Apache as a reverse proxy with an AJP connector in the server.xml file in a working post-JIRA 7.1 installation that now has a new client certificate requirement, although the use of separate VirtualHosts may also work with non-AJP connectors or even a different web server like Nginx.   If you are upgrading from an earlier version of JIRA you also may still first need to import the server SSL certificate into JIRA's Java trust store as noted in the Atlassian links in my original post above. 

[Jimmy Liberato]

Since Atlassian said they weren't going to fix the JIRA gadget title rendering bug and none of their proposed workarounds were relevant to the use of client certificates in a reverse proxy environment I decided I needed to attack this again.  I basically used Manse's suggestions above.   I created another virtual server in Apache to do the localhost part (after aliasing the FQDN part of the Base URL to 127.0.0.1 in the hosts file as above).  I just copy and pasted the entire <VirtualHost> block under the original and made a couple modifications between them.

So, the original Apache settings looked like this:

<VirtualHost *:443>

...all of the SSL, the certificates and AJP proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

I needed to restrict the VirtualHost to just the external IP address rather than all addresses (*) and create another VirtualHost to handle forwarding from the host file (which is checked and acted on before external DNS gets a chance to send any internal requests to the external IP address).  It now looks like this:

<VirtualHost [External IP Address goes here]:443>

...all of the SSL, the certificates and AJP  reverse proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

<VirtualHost 127.0.01:443>

...all of the SSL, the certificates and AJP reverse proxy configuration entries in the original but including:

SSLVerifyClient none

</VirtualHost>

Connections originating from the local system (JIRA talking to itself) now bypass the SSLVerifyClient requirement and the gadget title rendering bug does not manifest itself.

Note that this is only a solution for those using Apache as a reverse proxy with an AJP connector in the server.xml file in a working post-JIRA 7.1 installation that now has a new client certificate requirement, although the use of separate VirtualHosts may also work with non-AJP connectors or even a different web server like Nginx.   If you are upgrading from an earlier version of JIRA you also may still first need to import the server SSL certificate into JIRA's Java trust store as noted in the Atlassian links in my original post above. 

[Manse Wolken]

I did not remember that <VirtualHost [External IP Address goes here]:443> option for the Apache configuration.  

Well done, looks  good and is done with only one web server in the stack. Neat.

[Jimmy Liberato]

Since Atlassian said they weren't going to fix the JIRA gadget title rendering bug and none of their proposed workarounds were relevant to the use of client certificates in a reverse proxy environment I decided I needed to attack this again.  I basically used Manse's suggestions above.   I created another virtual server in Apache to do the localhost part (after aliasing the FQDN part of the Base URL to 127.0.0.1 in the hosts file as above).  I just copy and pasted the entire <VirtualHost> block under the original and made a couple modifications between them.

So, the original Apache settings looked like this:

<VirtualHost *:443>

...all of the SSL, the certificates and AJP proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

I needed to restrict the VirtualHost to just the external IP address rather than all addresses (*) and create another VirtualHost to handle forwarding from the host file (which is checked and acted on before external DNS gets a chance to send any internal requests to the external IP address).  It now looks like this:

<VirtualHost [External IP Address goes here]:443>

...all of the SSL, the certificates and AJP  reverse proxy configuration entries including:

SSLVerifyClient require

</VirtualHost>

<VirtualHost 127.0.01:443>

...all of the SSL, the certificates and AJP reverse proxy configuration entries in the original but including:

SSLVerifyClient none

</VirtualHost>

Connections originating from the local system (JIRA talking to itself) now bypass the SSLVerifyClient requirement and the gadget title rendering bug does not manifest itself.

Note that this is only a solution for those using Apache as a reverse proxy with an AJP connector in the server.xml file in a working post-JIRA 7.1 installation that now has a client certificate requirement.  If you are upgrading from an earlier version of JIRA you still must first do the import of the server SSL certificate into JIRA's Java trust store as noted in the Atlassian links in my original post above.