Nginx proxy 502 & 504 error, have to restart server every day

arturombravo

Hi, I have a linux server running confluence and jira. I installed nginx to create a reverse proxy for http://projects.domain.com (confluence) and http://projects.domain.com/issues (jira)

Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira.

Any ideas? ? ?

NGINX error.log

2015/09/21 12:15:10 [error] 1304#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GE$

2015/09/21 12:15:10 [error] 1304#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /favicon.ico HTT$

2015/09/21 12:22:52 [error] 1273#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLock$

2015/09/21 12:22:53 [error] 1273#0: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLoc$

 

 

Nginx/sites-enabled/

server {
    listen      80;
    server_name projects.wesuite.com;
    location / {
        proxy_pass  http://localhost:8090;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_connect_timeout   180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;

    }
    location /issues {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:8080/issues;

        proxy_connect_timeout 180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;
        client_max_body_size 25M;
    }

}

 

 

 

Confluence/conf/server.xml

<Server port="8000" shutdown="SHUTDOWN" debug="0">
    <Service name="Tomcat-Standalone">
        <Connector port="8090" connectionTimeout="20000" redirectPort="8443"
                maxThreads		="200" minSpareThreads="10"
                enableLookups	="false" acceptCount="10" debug="0" 	   
				URIEncoding		="UTF-8"
                protocol		="org.apache.coyote.http11.Http11NioProtocol" />

        <Engine name="Standalone" defaultHost="localhost" debug="0">

        <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false">

           <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
           <!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
                    <Manager pathname="" />
                </Context>
            </Host>


        </Engine>

 

 

 

Jira/conf/server.xml

 

 

<Service name="Catalina">
        <Connector port="8080"
                   maxThreads="150"
                   minSpareThreads="25"
                   connectionTimeout="20000"

                   enableLookups="false"
                   maxHttpHeaderSize="8192"
                   protocol="HTTP/1.1"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   acceptCount="100"
                   disableUploadTimeout="true"
                   proxyName="projects.wesuite.com" proxyPort="80" />

   <Engine name="Catalina" defaultHost="localhost">
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

   <Context path="/issues" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true">

   <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
                              factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
                    <Manager pathname=""/>
                </Context>


            </Host>

rrudnicki

Hi Arturo, 

 

Errors 5xx usually are related to some bad/missing configuration on the service/server side.

 

"Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira."

 

It seems your Tomcat (Confluence/JIRA) is more busy than your Nginx. When Nginx get some requests but Tomcat can't handle, you get '502 Bad Gateway', which meas Tomcat refused to work for Nginx. Error 504 is related to timeout.

Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', ‘ServerLimit'

Also check the memory from your servers (I’m talking about the memory of Confluence and JIRA, not the physical memory).

 

Regards, 

Renato Rudnicki