Nginx proxy 502 & 504 error, have to restart server every day

Hi, I have a linux server running confluence and jira. I installed nginx to create a reverse proxy for http://projects.domain.com (confluence) and http://projects.domain.com/issues (jira)

Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira.

Any ideas? ? ?


NGINX error.log

2015/09/21 12:15:10 [error] 1304#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GE$

2015/09/21 12:15:10 [error] 1304#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /favicon.ico HTT$

2015/09/21 12:22:52 [error] 1273#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLock$

2015/09/21 12:22:53 [error] 1273#0: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLoc$

 

 

Nginx/sites-enabled/

server {
    listen      80;
    server_name projects.wesuite.com;
    location / {
        proxy_pass  http://localhost:8090;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_connect_timeout   180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;

    }
    location /issues {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:8080/issues;

        proxy_connect_timeout 180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;
        client_max_body_size 25M;
    }

}

 

 

 

Confluence/conf/server.xml

<Server port="8000" shutdown="SHUTDOWN" debug="0">
    <Service name="Tomcat-Standalone">
        <Connector port="8090" connectionTimeout="20000" redirectPort="8443"
                maxThreads		="200" minSpareThreads="10"
                enableLookups	="false" acceptCount="10" debug="0" 	   
				URIEncoding		="UTF-8"
                protocol		="org.apache.coyote.http11.Http11NioProtocol" />

        <Engine name="Standalone" defaultHost="localhost" debug="0">

        <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false">

           <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true">
           <!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties -->
                    <Manager pathname="" />
                </Context>
            </Host>


        </Engine>

 

 

 

Jira/conf/server.xml

 

 

<Service name="Catalina">
        <Connector port="8080"
                   maxThreads="150"
                   minSpareThreads="25"
                   connectionTimeout="20000"

                   enableLookups="false"
                   maxHttpHeaderSize="8192"
                   protocol="HTTP/1.1"
                   useBodyEncodingForURI="true"
                   redirectPort="8443"
                   acceptCount="100"
                   disableUploadTimeout="true"
                   proxyName="projects.wesuite.com" proxyPort="80" />

   <Engine name="Catalina" defaultHost="localhost">
            <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

   <Context path="/issues" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true">

   <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction"
                              factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/>
                    <Manager pathname=""/>
                </Context>


            </Host>

3 answers

This widget could not be displayed.

Hi Arturo, 

 

Errors 5xx usually are related to some bad/missing configuration on the service/server side.

 

"Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira."

 

It seems your Tomcat (Confluence/JIRA) is more busy than your Nginx. When Nginx get some requests but Tomcat can't handle, you get '502 Bad Gateway', which meas Tomcat refused to work for Nginx. Error 504 is related to timeout.

Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', ‘ServerLimit'

Also check the memory from your servers (I’m talking about the memory of Confluence and JIRA, not the physical memory).

 

Regards, 

Renato Rudnicki

Hi @Renato Rudnicki Thanks for your help. After changing the Worker_connections and processes the 504 error is gone. However I am still getting a 502 error almost daily for confluence. Any idea what I may be missing ? ? ?

Hi Arturo, Can you let us know if this is happening in a specific time? I mean, when there's more users using Confluence/JIRA? Also, did you tried to increase the Confluence and JIRA memory (the application memory, not the server)?

If it helps at all, I was having this issue, too. I actually decreased my heap size which seemed to fix it (running for maybe one to two users on a smaller DigitalOcean machine).

Never thought to actually look at this (could never actually find in the logs), but I believe that the entire machine was running out of memory and hence the applications would randomly crash.

I am also facing the same issue but i dont think my nginx is running out of memory. I am using nginx as a proxy to connect to a back end service. @Renato Rudnicki what is the suitable number for 'worker_processes' and 'worker_connections'? I am getting same 504 bad gateway time out once in a while and restarting the nginx service works fine as expected. Unable to find the root cause. The back end service is working as expected and never had a downtime.  Appreciate your help here! Thanks in advance!

This widget could not be displayed.

Hi @Renato Rudnicki

I am also facing the below error:

502 Bad Gateway


nginx

my tomcat service is restarting every now and then, also its consuming high cpu and memory utilisation(fluctuating but mostly its above 150%).

$free -g     #fetched me the below results:

free-g.PNGCould you/anyone, please advice how could I resolve the issue.

Regards,

Aravind

You will need to look into the logs, and maybe get some thread traces when it is running with high cpu, so you can work out what it is doing.

Which log do I need to look into specifically?

Look for the files in <jira home>/logs, and <jira installation>/log

This widget could not be displayed.
X

Suggest an answer

Log in or Sign up to answer
Atlassian Summit 2018

Meet the community IRL

Atlassian Summit is an excellent opportunity for in-person support, training, and networking.

Learn more
Community showcase
Posted Aug 06, 2018 in Jira Service Desk

A is for Activate: Share your top Jira Service Desk onboarding tips for new users!

Hi, everyone! Molly here from the Jira Service Desk Product Marketing Team :).  In the spirit of this month's  august-challenge, we're sourcing stories of Jira Service Desk activation fro...

578 views 25 15
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you