Jira Software don't respond to request

[Samuele Pretini]

Hi,

I have installed Jira server self hosted on EC2 infrastructure, using M5D server type.

I have a starter license with 700 issue.

What happened: after a period of server up and running (1 month) when we access to Jira, the response time is very, very slow......the server is not responding. I have to restart the server and than all goes very very good, and the response time is good.

Why this happen ?

The memory and all the other parameter are setted as is described on the guide.

Jira server is: 7.12.3

Database: Mysql 5.7

[Nic Brough -Adaptavist- · Accepted solution]

You will need to read your application logs to find out why it starts to struggle and/or stops.

Check <jira home>/logs/atlassian-jira.log and then <jira installation>/logs/catalina.out - those two will tell you of crashes.  If it is slowing down and then stopping eventually, then it's more likely to be the first log, and you are probably looking for memory errors.  I'd want to monitor more carefully and capture data on memory usage.

[Samuele Pretini]

A small piece of catalina.out is this, but there isn't any message on memory:

java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:113)
at com.atlassian.plugin.cache.filecache.impl.OneStreamCache.stream(OneStreamCache.java:54)
at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.cache(FileCacheImpl.java:87)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCachedInProduction(Controller.java:319)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCached(Controller.java:282)
at com.atlassian.plugin.webresource.impl.http.Controller.serveResources(Controller.java:222)
at com.atlassian.plugin.webresource.impl.http.Controller.serveBatch(Controller.java:84)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:78)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:71)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.callHandler(BaseRouter.java:169)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.dispatch(BaseRouter.java:144)
at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:65)
at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:28)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
... 15 filtered
at com.develocenter.jira.extschemes.web.filter.DisableNewEntitiesFilter.doFilter(DisableNewEntitiesFilter.java:55)
... 3 filtered
at com.develocenter.jira.extschemes.web.filter.JIRALinkFilter.doFilter(JIRALinkFilter.java:70)
... 32 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 39 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 12 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 7 filtered
at com.atlassian.jira.plugin.webresource.CachingResourceDownloadRewriteRule$1.execute(CachingResourceDownloadRewriteRule.java:60)
at org.tuckey.web.filters.urlrewrite.RewrittenUrlClass.doRewrite(RewrittenUrlClass.java:61)
... 6 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 10 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 26 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356)
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:825)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:730)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
at com.atlassian.gzipfilter.GzipResponseStream.write(GzipResponseStream.java:46)
at com.atlassian.gzipfilter.RoutableServletOutputStream.write(RoutableServletOutputStream.java:134)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1384)
at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:110)
... 230 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)

2019-01-09 15:07:04,446 http-nio-8080-exec-9 ERROR [o.a.c.c.C.[.[localhost].[/].[noopservlet]] Servlet.service() for servlet [noopservlet] in context with path [] threw exception
java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:113)
at com.atlassian.plugin.cache.filecache.impl.OneStreamCache.stream(OneStreamCache.java:54)
at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.cache(FileCacheImpl.java:87)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCachedInProduction(Controller.java:319)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCached(Controller.java:282)
at com.atlassian.plugin.webresource.impl.http.Controller.serveResources(Controller.java:222)
at com.atlassian.plugin.webresource.impl.http.Controller.serveBatch(Controller.java:84)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:78)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:71)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.callHandler(BaseRouter.java:169)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.dispatch(BaseRouter.java:144)
at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:65)
at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:28)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
... 15 filtered
at com.develocenter.jira.extschemes.web.filter.DisableNewEntitiesFilter.doFilter(DisableNewEntitiesFilter.java:55)
... 3 filtered
at com.develocenter.jira.extschemes.web.filter.JIRALinkFilter.doFilter(JIRALinkFilter.java:70)
... 32 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 39 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 20 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 12 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 7 filtered
at com.atlassian.jira.plugin.webresource.CachingResourceDownloadRewriteRule$1.execute(CachingResourceDownloadRewriteRule.java:60)
at org.tuckey.web.filters.urlrewrite.RewrittenUrlClass.doRewrite(RewrittenUrlClass.java:61)
... 6 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 10 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 26 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356)
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:825)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:730)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:145)
at com.atlassian.gzipfilter.GzipResponseStream.write(GzipResponseStream.java:46)
at com.atlassian.gzipfilter.RoutableServletOutputStream.write(RoutableServletOutputStream.java:134)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1384)
at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:110)
... 230 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)

[Nic Brough -Adaptavist-]

Ok, I think you've hit one of the things in https://confluence.atlassian.com/jirakb/clientabortexception-java-io-ioexception-broken-pipe-225122378.html

[Samuele Pretini]

Another information: when the response doesn't arrive the CPU goes to the 90% and the machine is in starvation.

I have only to reboot Jira.

Than all goes very good, the CPU is on 5%

[Nic Brough -Adaptavist-]

Yes, that's consistent with something going wrong internally and being cleared out by a restart.  You'll need to work your way through the doc I pointed to (and follow the links)