Hi, I have a linux server running confluence and jira. I installed nginx to create a reverse proxy for http://projects.domain.com (confluence) and http://projects.domain.com/issues (jira)
Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira.
Any ideas? ? ?
NGINX error.log
2015/09/21 12:15:10 [error] 1304#0: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GE$
2015/09/21 12:15:10 [error] 1304#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /favicon.ico HTT$
2015/09/21 12:22:52 [error] 1273#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLock$
2015/09/21 12:22:53 [error] 1273#0: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 71.167.124.112, server: projects.wesuite.com, request: "GET /issues/JiraLoc$
Nginx/sites-enabled/
server { listen 80; server_name projects.wesuite.com; location / { proxy_pass http://localhost:8090; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 180; proxy_send_timeout 180; proxy_read_timeout 180; } location /issues { proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://localhost:8080/issues; proxy_connect_timeout 180; proxy_send_timeout 180; proxy_read_timeout 180; client_max_body_size 25M; } }
Confluence/conf/server.xml
<Server port="8000" shutdown="SHUTDOWN" debug="0"> <Service name="Tomcat-Standalone"> <Connector port="8090" connectionTimeout="20000" redirectPort="8443" maxThreads ="200" minSpareThreads="10" enableLookups ="false" acceptCount="10" debug="0" URIEncoding ="UTF-8" protocol ="org.apache.coyote.http11.Http11NioProtocol" /> <Engine name="Standalone" defaultHost="localhost" debug="0"> <Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false"> <Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"> <!-- Logger is deprecated in Tomcat 5.5. Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --> <Manager pathname="" /> </Context> </Host> </Engine>
Jira/conf/server.xml
<Service name="Catalina"> <Connector port="8080" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false" maxHttpHeaderSize="8192" protocol="HTTP/1.1" useBodyEncodingForURI="true" redirectPort="8443" acceptCount="100" disableUploadTimeout="true" proxyName="projects.wesuite.com" proxyPort="80" /> <Engine name="Catalina" defaultHost="localhost"> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true"> <Context path="/issues" docBase="${catalina.home}/atlassian-jira" reloadable="false" useHttpOnly="true"> <Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction" factory="org.objectweb.jotm.UserTransactionFactory" jotm.timeout="60"/> <Manager pathname=""/> </Context> </Host>
Hi Arturo,
Errors 5xx usually are related to some bad/missing configuration on the service/server side.
"Everything seems to work fine, however at least 1x day I get either a 502, or 504 error. Trying to ssh into the server will also sometimes fail, and the only way to resolve is to restart the server. and do a service restart for nginx,confluence, and jira."
It seems your Tomcat (Confluence/JIRA) is more busy than your Nginx. When Nginx get some requests but Tomcat can't handle, you get '502 Bad Gateway', which meas Tomcat refused to work for Nginx. Error 504 is related to timeout.
Try decrease 'worker_connections' and 'worker_processes' in Nginx and increase 'MaxClients', ‘ServerLimit'
Also check the memory from your servers (I’m talking about the memory of Confluence and JIRA, not the physical memory).
Regards,
Renato Rudnicki
Hi @Renato Rudnicki Thanks for your help. After changing the Worker_connections and processes the 504 error is gone. However I am still getting a 502 error almost daily for confluence. Any idea what I may be missing ? ? ?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi Arturo, Can you let us know if this is happening in a specific time? I mean, when there's more users using Confluence/JIRA? Also, did you tried to increase the Confluence and JIRA memory (the application memory, not the server)?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
If it helps at all, I was having this issue, too. I actually decreased my heap size which seemed to fix it (running for maybe one to two users on a smaller DigitalOcean machine).
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Never thought to actually look at this (could never actually find in the logs), but I believe that the entire machine was running out of memory and hence the applications would randomly crash.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I am also facing the same issue but i dont think my nginx is running out of memory. I am using nginx as a proxy to connect to a back end service. @Renato Rudnicki what is the suitable number for 'worker_processes' and 'worker_connections'? I am getting same 504 bad gateway time out once in a while and restarting the nginx service works fine as expected. Unable to find the root cause. The back end service is working as expected and never had a downtime. Appreciate your help here! Thanks in advance!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @rrudnicki,
I am also facing the below error:
nginx
my tomcat service is restarting every now and then, also its consuming high cpu and memory utilisation(fluctuating but mostly its above 150%).
$free -g #fetched me the below results:
Could you/anyone, please advice how could I resolve the issue.
Regards,
Aravind
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You will need to look into the logs, and maybe get some thread traces when it is running with high cpu, so you can work out what it is doing.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Look for the files in <jira home>/logs, and <jira installation>/log
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.