Gadgets are showen like __MSG_gadget after setting a apache ssl proxy

Sascha Thiel August 16, 2016

Hi,

I'm using JIRA with https proxied by apache. I was following the steps from this link.

https://confluence.atlassian.com/adminjiraserver071/integrating-jira-with-apache-using-ssl-802593043.html

Almost everything works fine, but the names of the gadgets are shown like __MSG_gadget... and the connection to confluence does not work propertly. I searched what the problem could be, but the only thing I found was an SSLHandshake exception what usually happens when using self signed certificates. This is a bit supprising, because I got my certificate signed by an official CA.

Followed you can find my configs.

server.xml

<Connector port="8101"
	maxThreads="150"
	minSpareThreads="25"
	connectionTimeout="20000"
	enableLookups="false"
	maxHttpHeaderSize="8192"
	protocol="HTTP/1.1"
	useBodyEncodingForURI="true"
	redirectPort="8443"
	acceptCount="100"
	disableUploadTimeout="true"
	scheme="https"
	proxyName="sub.domain.tld"
	proxyPort="443"/>
	
<Connector port="8100"
   maxThreads="150"
   minSpareThreads="25"
   connectionTimeout="20000"
   enableLookups="false"
   maxHttpHeaderSize="8192"
   protocol="HTTP/1.1"
   useBodyEncodingForURI="true"
   redirectPort="8443"
   acceptCount="100"
   disableUploadTimeout="true"/>

vhosts

<VirtualHost xx.xx.xx.xx:443>
	ServerAdmin admin@domain.tld
	ServerName sub.domain.tld
	SSLEngine on
	SSLCertificateFile /etc/ssl/domain_tld.crt
	SSLCertificateKeyFile /etc/ssl/domain_tld.key
	SSLCertificateChainFile /etc/ssl/ca_bundle.crt
	ProxyRequests Off
	ProxyVia Off
	<Proxy *>
			Order Deny,Allow
			Allow from all
	</Proxy>
	ProxyPass / http://localhost:8101/
	ProxyPassReverse / http://localhost:8101/
	ErrorLog ${APACHE_LOG_DIR}/issues-error.log
	CustomLog ${APACHE_LOG_DIR}/issues-access.log combined
</VirtualHost>

Do you have any Idea what I'm doing wrong and if the issues are related to each other or are this two different issues?

Thanks in advance,
Sascha

3 answers

1 accepted

0 votes
Answer accepted
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
August 16, 2016

The configs there look ok, but the error is usually caused by JIRA not being able to talk to itself.  Imagine that the dashboard is nothing to do with JIRA itself, but needs to read JIRA in order to work - you have to provide a valid route from the dashboard to JIRA. 

So, have you imported the ssl keys required into the keystore?

Sascha Thiel August 16, 2016

Hi Nic,

Thanks for checking the configs.

No, I haven't imported the ssl keys yet, because I thought this is only neseccary for selfsigned certificates (mine is signed by an trusted CA). Isn't this the case? If not, can you explain me why?

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
August 16, 2016

The JVM is trying to read a url, which you've protected with certificate checks.  It can't get to the url, because it doesn't have the certificates.

Self signing is irrelevant, it needs the certificate to present to the url!

Sascha Thiel August 16, 2016

I added now both crt files to the keystore but the issue is still there.

keytool -import -trustcacerts -keystore ../lib/security/cacerts -storepass changeit -noprompt -alias {alias} -file /etc/ssl/domain_tld.crt

 

 

Sascha Thiel August 17, 2016

Any further ideas what the issue could be?

Thanks in advance

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
August 17, 2016

I'm afraid it could be anything, I've just gone over the issues I've had in the past, but that's not exhaustive - you need to probe how your server gets to talk to itself, and why it can't fetch things from itself over a REST connection.

Sascha Thiel August 17, 2016

I tried also

# java SSLPoke sub.domain.tld 443
# Successfully connected

described here: https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-779355358.html

but with

# java httpclienttest-1.0.jar sub.domain.tld:443

described here: https://bitbucket.org/atlassianlabs/httpclienttest

I get this execption

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at com.atlassianlabs.sslclient.Main.main(Main.java:27)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
        ... 18 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
        ... 24 more
Exception in thread "main" java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at com.atlassianlabs.sslclient.Main.main(Main.java:32)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at com.atlassianlabs.sslclient.Main.main(Main.java:27)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
        ... 18 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
        ... 24 more
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
August 17, 2016

Your certificates are wrong then.

oops, hit enter too quickly, sorry.

Are you sure your client certificates match?  More importantly, have you imported them into the correct keystore - a lot of machines have Java installed, but JIRA may not be using the installed Java you expect it to.

Sascha Thiel August 18, 2016

Hi, couldn't answer yerterday due to 24h restrictions...

Are you sure your client certificates match?

I hope so. The browser isn't complaining or do I get something wrong? So followed the steps I did.

  1. Ordered a wildcard ssl certificate for my domain.
  2. Copied the certificates and the key files to the server.
  3. Added the ssl apache conf like above.
  4. Changed the connector for jira.
  5. Imported the certificates into the keystore like above.

Did I forgot something or have done something wrong here?

More importantly, have you imported them into the correct keystore - a lot of machines have Java installed, but JIRA may not be using the installed Java you expect it to.

I used the keytool from the jre with comes with JIRA. I think this is the one used by JIRA due to the console log when I start JIRA.

Using JRE_HOME: /opt/atlassian/jira/jre/

But yesterday I tested the connection with httpclient also from my local machine (same error), so now I think it is more an issue of my apache ssl setup rather than with JIRA itself.

Thanks again for your help. I'm getting crazy with this issue.

Sascha Thiel August 18, 2016

Hey, me again.

In the meanwhile I could fix the issue.

I had to explicitly set verify clieant to none.

SSLVerifyClient none

Now all is working as expected.

A Big thanks for your support on that issue.

chetan patil May 9, 2017

Hi Sascha,

We are also facing the same issue with gadgets after upgrade JIRA from 6.3.15 to 7.2.7.

Could you please let us know where did you add "SSLVerifyClient none" line exactly.

Regards,

 

0 votes
Vickey Palzor Lepcha
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 12, 2018

I've had this issue in the past - reimporting the SSL Certificate did resolve my issue.

0 votes
chetan patil May 9, 2017

Hi Sascha,

We are also facing the same issue with gadgets after upgrade JIRA from 6.3.15 to 7.2.7.

Could you please let us know where did you add "SSLVerifyClient none" line exactly.

 

Regards,

 

sascha_thiel May 18, 2017

Hey, I added it into the VirtualHost definition.

<VirtualHost xx.xx.xx.xx:443>

... something

SSLEngine on
SSLVerifyClient none
... certificate stuff

... something

</VirtualHost>
JP August 22, 2017

I am having the same issue, but in my case is beacuse I have been directed to to use SSLVerifyClient require. Once I chage from optional to require,  it brakes  the filters, gadgets and AppLink. Do you know if there is any workround if i have set apache to SSLVerifyClient to require?

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
August 22, 2017

That flag makes Apache require a valid client certificate.  You'll need to create a client cert for the server and enable it in the JVM so that it can talk to itself over SSL.

Jimmy Liberato March 12, 2018

Nic, your comment gives hope to my problem where SSLVerifyClient require breaks the gadget title rendering on JIRA 7.1+ (works fine without that requirement)

I can figure out how to generate a client certificate and how to make the JVM trust it but how do I make JIRA offer it up when requested by Apache?  Any ideas on that?

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
March 12, 2018

It happens automatically when the certificate is asked for, the JVM handles it.

Suggest an answer

Log in or Sign up to answer