Development Panel does not work through proxy

Question

The AppLink configuration between Stash and JIRA appears to create without issue. Both systems use mod_proxy through an Apache reverse proxy to translate the external address https://server.example.com to https://server-backend.example.org. On each of the back end systems mod_proxy, again, translates https://server-backend.example.org to http://server-backend.example.org:8080 (in the case of Jira) and http://server-backend.example.org:7990 (in the case of Stash). With the Applications configured this way and the AppLink configured to use the external addresses, the Link works as expected everywhere except on issues where the Development Panel would normally show the Commit/Pull request information on the right hand side. Instead it is showing the "Source" tab and the information in the tab shows as expected. If I configure the servers to run outside of the proxy (e.g. you connect directly to http://server-backend.example.org:8080) and I create the AppLink using the direct, non-ssl, non-proxy address, the Development panel functions as expected. Below I've provided the the portion of each servers server.xml file as well as the apache configurations for each system on the reverse proxy server. JIRA server.xml: Stash server.xml: JIRA apache configuration: NameVirtualHost server.example.com:443 ServerName server.example.com ErrorLog logs/server_ssl_error_log TransferLog logs/server_ssl_access_log LogLevel warn SSLEngine on SSLProtocol all -SSLv2 SSLHonorCipherOrder On SSLCipherSuite -All:EDH-DSS-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:RC4-MD5:RC4-SHA:DHE-DSS-AES128-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:AES128-SHA:AES256-SHA SSLCertificateFile /etc/pki/tls/certs/server.crt SSLCertificateKeyFile /etc/pki/tls/private/server.key SSLCertificateChainFile /etc/pki/tls/certs/incommon.crt SSLCACertificateFile /etc/pki/tls/certs/bundle.crt SSLVerifyClient optional ErrorDocument 403 "You are not allowed to access this site. Please ensure you are on a trusted network" Order deny,allow Deny from all # All private subnets Allow from 10.0.0.0/8 # All MALNET VPN Allow from xxx.xxx.xxx.xxx # All TESTNET VPN Allow from xxx.xxx.xxx.xxx # All Workstations Allow from xxx.xxx.xxx.0/24 # All DEV IPs Allow from xxx.xxx.xxx.0/24 Order deny,allow Deny from all # All private subnets Allow from 10.0.0.0/8 # All MALNET VPN Allow from xxx.xxx.xxx.xxx # All TESTNET VPN Allow from xxx.xxx.xxx.xxx # All Workstations Allow from xxx.xxx.xxx.0/24 # All DEV IPs Allow from xxx.xxx.xxx.0/24 SSLProxyEngine On ProxyRequests On ProxyPreserveHost On ProxyPass /cert_info.php ! ProxyPass / https://10.xxx.xxx.228:443/ retry=0 connectionTimeout=5 timeout=300 ProxyPassReverse / http://10.xxx.xxx.228:443/ retry=0 ProxyTimeout 300 # Only require certificates for some projects SSLOptions +StdEnvVars RewriteEngine On # git (fisheye triggers) RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.80$ # spud Is this still needed? RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.109$ # wiki RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.167$ # wiki-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.184$ # jira-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.12$ # scm-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.228$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.252$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.253$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.251$ # xxxxxxxxx for jenkins RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.68$ RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$ RewriteRule .* /cert_info.php [R,L] SSLOptions +StdEnvVars SSLVerifyDepth 10 SSLOptions +StdEnvVars SSLOptions +StdEnvVars SetEnvIf User-Agent ".*MSIE.*" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 CustomLog logs/ssl_request_log \ "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" JIRA Apache Config: NameVirtualHost jira.example.com:443 SSLStrictSNIVHostCheck off ServerName jira.example.com ErrorLog logs/ssl_error_log TransferLog logs/ssl_access_log LogLevel warn SSLEngine on SSLProtocol all -SSLv2 SSLHonorCipherOrder On SSLCipherSuite -All:EDH-DSS-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA:RC4-MD5:RC4-SHA:DHE-DSS-AES128-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:AES128-SHA:AES256-SHA SSLCertificateFile /etc/pki/tls/certs/jira.crt SSLCertificateKeyFile /etc/pki/tls/private/jira.key SSLCertificateChainFile /etc/pki/tls/certs/incommon.crt SSLCACertificateFile /etc/pki/tls/certs/bundle.crt SSLVerifyClient optional Order deny,allow Allow from all Order deny,allow Deny from all Allow from all SSLProxyEngine On ProxyRequests On ProxyPreserveHost on # Jira-be ProxyPass /cert_info.php ! ProxyPass / https://10.xxx.xxx.12:443/ retry=0 ProxyPassReverse / http://10.xxx.xxx.12:443/ retry=0 ProxyTimeout 300 SSLOptions +StdEnvVars RewriteEngine On # spud RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.109$ # wiki RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.167$ # wiki-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.184$ # jira-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.12$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.251$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.252$ RewriteCond %{REMOTE_ADDR} !^xxx\.xxx\.xxx\.253$ # scm-be RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.228$ # RT # 3790 allow xxxxxxx-xxxxxx RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.95$ # Jama RewriteCond %{REMOTE_ADDR} !^10\.xxx\.xxx\.192$ RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$ RewriteRule .* /cert_info.php [R,L] SSLOptions +StdEnvVars SSLVerifyDepth 10 SSLOptions +StdEnvVars SSLOptions +StdEnvVars SetEnvIf User-Agent ".*MSIE.*" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 CustomLog logs/ssl_request_log \ "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" I have my suspicions as to why this is not working but I'm not familiar enough with how the two applications communicate over the AppLink connection to know where to begin. There are no obvious errors in either the Stash or JIRA logs and all other features of the AppLink appear to work just fine. Any suggestions would be greatly appreciated.

Dave C · Answer

It's possible you could be running into the bug we have tracked in https://jira.atlassian.com/browse/JRA-40968 - there are a number of workarounds listed in there you could try.

Ted W · Answer

Quick update. I re-configured the proxy to use port 80 and disabled ssl for Stash, leaving JIRA to proxy over https and the development panel appears to work now.... Still not sure why this wouldn't work over an https reverse proxy.

Once I figured the above information out I re-enabled https on Stash. I then created a p12 file from the stash server certificate, key and CA root and imported that as alias "tomcat" in /usr/lib/jvm/java/jre/lib/security/cacerts, ensured that the proxy server also had the proper cert/key and CA specified in the apache config and none of this seemed to resolve the issue. Any other ideas why ssl proxy would be causing the development panel to malfunction would be appreciated.

Ted W · Answer

I just checked and no, it does not change the behavior. I also tried setting up the "Application Link" using the direct http://jira.example.org:8080 and http://stash.example.org:7990/git urls, leaving the respective applications' Base URL set to the proxy URL and it connected ok, once again and the Development panel now shows up with a message: "JIRA is having difficulty contacting Stash. If this condition persists, please contact your JIRA administrators." and the Source tab is also visible with the message: "Request to https://stash.example.com/git failed: Stash returned an error: 401 Unauthorized" which I found odd... not sure if it's useful information or not, though.