Sporadic WebGUI failure

Kale Bullen

Recently I have come across a sporadic issue that is causing a Jira Server webGUI to not resolve.

This system is running NGINX.

I have checked all running services to ensure that all services are running when the system ends up in this state.

I am unable to locate the cause of the issue. Could someone please point me in the right direction to what is causing this issue?

A simple reboot brings it all back up on line.

Daniel Eads

Hi Kale,

There are many, many possibilities. Let's try to gather more information to see if we can narrow them down!

1. When this problem comes up, what are the specific symptoms? Example: do you get an error in the browser, or does the browser window simply appear to be loading (and only give you a white page) for 60+ seconds without actually stopping its attempt to load?
2. Do you have other sites configured (such as a test page or error page) in nginx that you can use to confirm that nginx has not hung?
3. Do any errors appear in Jira's log file when this occurs?
4. Do you have nginx logging enabled, and do the nginx logs show that nginx responded to access requests during the timeframe this happens?

Some general thoughts:

• nginx has a value you can set in the server block for your reverse proxy to stop trying to connect after a bit and instead throw an nginx error. You can try adding this in your server block to rule out problems with nginx:

  proxy_read_timeout 60s;

• Particular Jira plugins could cause severe performance issues that might show symptoms like this. Have you recently upgraded Jira or any plugins that might coincide with the time you started noticing this behavior?

I think possibly the most likely scenario (without additional information) is that your Java heap for Jira may be undersized, it's using the CMS garbage collector, and what you're seeing is the JVM trying (unsuccessfully) to complete a full garbage collection. You could confirm this by looking at the GC logs which should be in <jira's install directory>/logs. A tool like GCViewer would be helpful to analyze your GC logs if you're not familiar with Java's Garbage Collection mechanisms. I would be interested in knowing the following related to this theory:

1. What are your current maximum and minimum heap values? (if you're not sure what these are, I'd also settle for the "Total Memory" value in Jira's System Information page):
   
2. Have you configure Jira to use G1GC? (you can search the System Information page for "G1GC" - if your browser can't find it on this page, the answer is "no")
3. Does GCViewer show your heap usage creeping up and stopping, or can you see a Full GC event in the logs around the time things seem to stop working?

Finally, I would just mention that restarting the services individually (Jira and nginx) rather than doing a full system reboot is going to help you narrow down where the cause of this is. If possible, eliminate variables by only changing one thing at once.

Cheers,
Daniel