Nginx proxy 502 & 504 error, have to restart server every day

arturombravo September 22, 2015

Hi, I have a linux server running confluence and jira. I installed nginx to create a reverse proxy for http://projects.domain.com (confluence) and http://projects.domain.com/issues (jira)

Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira.

Any ideas? ? ?


NGINX error.log

2015/09/21 12:15:10 [error] 1304#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GE$

2015/09/21 12:15:10 [error] 1304#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /favicon.ico HTT$

2015/09/21 12:22:52 [error] 1273#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLock$

2015/09/21 12:22:53 [error] 1273#0: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLoc$

 

 

Nginx/sites-enabled/

server {
    listen      80;
    server_name projects.wesuite.com;
    location / {
        proxy_pass  http://localhost:8090;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_connect_timeout   180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;

    }
    location /issues {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:8080/issues;

        proxy_connect_timeout 180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;
        client_max_body_size 25M;
    }

}

 

 

 

Confluence/conf/server.xml

<Server port="8000" shutdown="SHUTDOWN" debug="0">
    <Service name="Tomcat-Standalone">
        <Connector port="8090" connectionTimeout="20000" redirectPort="8443"
                maxThreads		="200" minSpareThreads="10"
                enableLookups	="false" acceptCount="10" debug="0" 	   
				URIEncoding		="UTF-8"
                protocol		="org.apache.coyote.http11.Http11NioProtocol" />

        <Engine name="Standalone" defaultHost="localhost" debug="0">

        <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false">

           <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
           <!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
                    <Manager pathname="" />
                </Context>
            </Host>


        </Engine>

 

 

 

Jira/conf/server.xml

 

 

<Service name="Catalina">
        <Connector port="8080"
                   maxThreads="150"
                   minSpareThreads="25"
                   connectionTimeout="20000"

                   enableLookups="false"
                   maxHttpHeaderSize="8192"
                   protocol="HTTP/1.1"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   acceptCount="100"
                   disableUploadTimeout="true"
                   proxyName="projects.wesuite.com" proxyPort="80" />

   <Engine name="Catalina" defaultHost="localhost">
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

   <Context path="/issues" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true">

   <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
                              factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
                    <Manager pathname=""/>
                </Context>


            </Host>

3 answers

1 vote
rrudnicki
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 22, 2015

Hi Arturo, 

 

Errors 5xx usually are related to some bad/missing configuration on the service/server side.

 

"Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira."

 

It seems your Tomcat (Confluence/JIRA) is more busy than your Nginx. When Nginx get some requests but Tomcat can't handle, you get '502 Bad Gateway', which meas Tomcat refused to work for Nginx. Error 504 is related to timeout.

Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', ‘ServerLimit'

Also check the memory from your servers (I’m talking about the memory of Confluence and JIRA, not the physical memory).

 

Regards, 

Renato Rudnicki

arturombravo September 28, 2015

Hi @Renato Rudnicki Thanks for your help. After changing the Worker_connections and processes the 504 error is gone. However I am still getting a 502 error almost daily for confluence. Any idea what I may be missing ? ? ?

rrudnicki
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 28, 2015

Hi Arturo, Can you let us know if this is happening in a specific time? I mean, when there's more users using Confluence/JIRA? Also, did you tried to increase the Confluence and JIRA memory (the application memory, not the server)?

Jeff Stubler August 31, 2016

If it helps at all, I was having this issue, too. I actually decreased my heap size which seemed to fix it (running for maybe one to two users on a smaller DigitalOcean machine).

Jeff Stubler November 26, 2016

Never thought to actually look at this (could never actually find in the logs), but I believe that the entire machine was running out of memory and hence the applications would randomly crash.

Roshan Paul November 29, 2016

I am also facing the same issue but i dont think my nginx is running out of memory. I am using nginx as a proxy to connect to a back end service. @Renato Rudnicki what is the suitable number for 'worker_processes' and 'worker_connections'? I am getting same 504 bad gateway time out once in a while and restarting the nginx service works fine as expected. Unable to find the root cause. The back end service is working as expected and never had a downtime.  Appreciate your help here! Thanks in advance!

0 votes
ARAVIND MURALI May 12, 2018
X
0 votes
ARAVIND MURALI May 12, 2018

Hi @rrudnicki

I am also facing the below error:

502 Bad Gateway


nginx

my tomcat service is restarting every now and then, also its consuming high cpu and memory utilisation(fluctuating but mostly its above 150%).

$free -g     #fetched me the below results:

free-g.PNGCould you/anyone, please advice how could I resolve the issue.

Regards,

Aravind

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
May 12, 2018

You will need to look into the logs, and maybe get some thread traces when it is running with high cpu, so you can work out what it is doing.

ARAVIND MURALI May 12, 2018

Which log do I need to look into specifically?

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
May 12, 2018

Look for the files in <jira home>/logs, and <jira installation>/log

Suggest an answer

Log in or Sign up to answer