Jira 503 Service Unavailable w/ Apache SSL Proxy

Tito Valentin September 8, 2015

After restarting my server (I run JIRA and Confluence on the same host), JIRA is giving me a 503 Service Unavailable error on the browser. I can telnet to port 8080 just fine on localhost and remotely so the service is running. I am using Apache Reverse Proxy and have added the retry=0 timeout=5 to my ProxyPass option in my httpd config file. I still cannot access Jira. I am running Apache 2.4

Confluence (on the same host) is configured exactly the same way and I can connect to it just fine. I also don't have SELinux running. It is completely disabled. I even tried with the firewall completely open and have the same results.

My Apache Config file for JIRA is as follows:

## Apache Config:

 

<VirtualHost *:80>

        ServerAdmin webmaster@domain.tld

        ServerName support.domain.tld

        ServerAlias support.domain.tld

        Redirect permanent "/" "https://support.domain.tld/"

</VirtualHost>

 

## SSL Virtual Host Context

<VirtualHost *:443>

        ServerAdmin webmaster@domain.tld

        ServerName support.domain.tld

        ServerAlias support.domain.tld

        SSLEngine               on

        SSLCertificateFile      /etc/pki/tls/certs/wildcard.domain.tld/wildcard.domain.tld.crt

        SSLCertificateKeyFile   /etc/pki/tls/certs/wildcard.domain.tld/wildcard.domain.tld.key

        SSLCACertificateFile    /etc/pki/tls/certs/wildcard.domain.tld/intermediate.crt

        ErrorLog                /var/log/httpd/jira.error.log

        LogLevel                debug

        CustomLog               /var/log/httpd/jira.access.log combined

        ## Proxy Settings

        SSLProxyEngine          On

        ProxyRequests           Off

        ProxyPreserveHost       On

        ProxyPass               /       http://support.domain.tld:8080/ retry=0 connectiontimeout=600 timeout=600 nocanon

        ProxyPassReverse        /       http://support.domain.tld:8080/

        <Proxy *>

                #Order deny,allow

                #Allow from all

                Require all granted

        </Proxy>

</VirtualHost>

My JIRA server.xml file contains the following connector settings:

<Connector acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" maxHttpHeaderSize="8192"maxThreads="150" minSpareThreads="25" port="8080" protocol="HTTP/1.1" redirectPort="8443" useBodyEncodingForURI="true" scheme="https"proxyName="support.domain.tld" proxyPort="443" secure="true"/>

My Apache error logs for JIRA complains about the following:

[Tue Sep 08 07:15:14.306959 2015] [proxy:debug] [pid 32575] proxy_util.c(2215): AH00943: HTTP: has released connection for (support.domain.tld)

[Tue Sep 08 07:15:14.307005 2015] [headers:debug] [pid 32575] mod_headers.c(848): AH01503: headers: ap_headers_error_filter()

[Tue Sep 08 07:15:14.307135 2015] [ssl:debug] [pid 32575] ssl_engine_io.c(992): [client XXX.XXX.XXX.XXX:46247] AH02001: Connection closed to child 1 with standard shutdown (server support.domain.tld:443)

[Tue Sep 08 07:15:15.202863 2015] [proxy:error] [pid 31239] (110)Connection timed out: AH00957: HTTP: attempt to connect to XXX.XXX.XXX.XXX:8080 (support.domain.tld) failed

[Tue Sep 08 07:15:15.202918 2015] [proxy:error] [pid 31239] AH00959: ap_proxy_connect_backend disabling worker for (support.domain.tld) for 0s

[Tue Sep 08 07:15:15.202925 2015] [proxy_http:error] [pid 31239] [client XXX.XXX.XXX.XXX:46990] AH01114: HTTP: failed to make connection to backend: support.domain.tld

Interesting enough, my Apache error logs for Confluence have the same errors but I can get to Confluence just fine. Keep in mind that my Apache config for Confluence and the Connector setting for Confluence in server.xml are the same as Jira. I don't know why JIRA stopped working and Confluence works fine. I even stopped the Confluence server to see if one they were competing against each other but I have the same results.

When I run a wget from the Linux CLI to my support.domain.tld, I get the following:

wget --spider support.domain.tld

Spider mode enabled. Check if remote file exists.

--2015-09-07 20:59:35--  http://support.domain.tld/

Resolving support.domain.tld (support.domain.tld)... XXX.XXX.XXX.XXX

Connecting to support.domain.tld (support.domain.tld|XXX.XXX.XXX.XXX|:80... connected.

HTTP request sent, awaiting response... 301 Moved Permanently

Location: https://support.domain.tld/ [following]

Spider mode enabled. Check if remote file exists.

--2015-09-07 20:59:35--  https://support.domain.tld/

Connecting to support.domain.tld (support.domain.tld)|XXX.XXX.XXX.XXX|:443... connected.

HTTP request sent, awaiting response... 503 Service Unavailable

Remote file does not exist -- broken link!!!

What am I missing here? This was working fine and I only rebooted the server to add more RAM.

2 answers

1 accepted

1 vote
Answer accepted
Tito Valentin September 8, 2015

I was able to solve the problem by changing the following from:

ProxyRequests           Off
ProxyPreserveHost       On
ProxyPass               /       http://support.domain.tld:8080/ retry=0 connectiontimeout=600 timeout=600 nocanon
ProxyPassReverse        /       http://support.domain.tld:8080/

To:

ProxyRequests           Off
ProxyPreserveHost       Off
ProxyPass               /       http://localhost:8080/ retry=0 connectiontimeout=600 timeout=600 nocanon
ProxyPassReverse        /       http://localhost:8080/

It prefers localhost instead of the actual domain. I also changed ProxyPreserveHost from On to Off so that it uses what I specify in my ProxyPass line instead of what I request coming in via the browser.

One other thing I added was the Location directive right after my Proxy directive:

<Proxy *>
    Require all granted
</Proxy>

<Location />
    Require all granted
</Location>

My server.xml looks like this for the Connector directive:

<Connector acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" maxHttpHeaderSize="8192" maxThreads="150" minSpareThreads="25" port="8080" protocol="HTTP/1.1" useBodyEncodingForURI="true" scheme="https" redirectPort="8443" proxyName="support.domain.tld" proxyPort="443" secure="true"/>
2 votes
rrudnicki
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 8, 2015

Hi Tito, 

 

Can you confirm if you are running Confluence over SSL as well? If so, I don’t believe this will works on the same server than JIRA since looks like you are using a wildcard certificate.

 

If you are running Confluence with port 443 enabled, this probably is the cause of you are getting the error 503. Try the following:

Stop JIRA

Run netstat -vant |grep 443 |grep -i listen  and check if there’s some port 443 up. If so, you are having conflict.  

 

Firewall shouldn’t be an issue, since this shouldn’t block loopback address (unless you setup that… ).

 

The best way to troubleshoot this is. Make sure everything works, one-by-one. I suggest you to try this (if it is possible):

Disable the SSL in both Confluence and JIRA and make sure they are running fine through the proxy. If you achieve this, than, enable SSL on JIRA (with no proxy) and see if this works. If so, than try to bypass through the proxy. On this way will will exactly see where are you facing issues and will be much easier to find the culprit and fix that. 

 

Hope this helps :).

 

Cheers, 

Renato Rudnicki

Tito Valentin September 8, 2015

Thanks for the pointers, Renato! stopped both JIRA and Confluence and did see 443 still running but that should be fine because Apache is listening on both 80 and 443, which is what I am using for Proxy:

 

[root@hostname ~]# ps -eaf | egrep -i "jira|confluence" | grep -v grep

[root@hostname ~]# netstat -vant | grep 443 | grep -i listen

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

I am going to try the other suggestions you made now and see where that takes me but wanted to show you that 443 is still listening due to Apache. I'll post more info shortly.

Suggest an answer

Log in or Sign up to answer