crowd application is down and after some time it is up and running so what might be route cause

[thejeswarareddy_r]

Without any manual interrupt application is back in online within 30 minutes so we would like find the route cause of crowd application down.

When we looked into application logs below we found something fishy

java.lang.RuntimeException: com.atlassian.plugin.servlet.DownloadException: Unable to stream to the output
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$6.writeTo(ResourceServingHelpers.java:360)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.lambda$writeTo$0(ResourceServingHelpers.java:184)
at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$2.writeTo(ResourceServingHelpers.java:184)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$8.writeTo(ResourceServingHelpers.java:384)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$1.writeTo(ResourceServingHelpers.java:125)
at com.atlassian.plugin.webresource.impl.http.Controller$6.write(Controller.java:362)
at com.atlassian.plugin.cache.filecache.impl.PassThroughCache.cache(PassThroughCache.java:18)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCachedInProduction(Controller.java:359)
at com.atlassian.plugin.webresource.impl.http.Controller.sendCached(Controller.java:322)
at com.atlassian.plugin.webresource.impl.http.Controller.serveResources(Controller.java:262)
at com.atlassian.plugin.webresource.impl.http.Controller.serveBatch(Controller.java:124)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:78)
at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:71)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.callHandler(BaseRouter.java:169)
at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.dispatch(BaseRouter.java:144)
at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:65)
at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:28)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:392)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
at com.atlassian.crowd.util.ResourceDownloadRewriteRule$1.execute(ResourceDownloadRewriteRule.java:56)
at org.tuckey.web.filters.urlrewrite.RewrittenUrlClass.doRewrite(RewrittenUrlClass.java:61)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.plugin.web.filter.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:27)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.security.BlockOpenSocialRequestFilter.doFilter(BlockOpenSocialRequestFilter.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.console.filter.CrowdCachingFilter.doFilter(CrowdCachingFilter.java:25)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.plugin.web.filter.NodeInformationFilter.doFilterInternal(NodeInformationFilter.java:37)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at com.atlassian.crowd.console.filter.CrowdDelegatingFilterProxy.doFilter(CrowdDelegatingFilterProxy.java:36)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.plugin.web.filter.HttpRequestLoggingFilter.doFilter(HttpRequestLoggingFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.crowd.plugin.web.filter.HttpContextFilter.doFilter(HttpContextFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.atlassian.plugin.servlet.DownloadException: Unable to stream to the output
at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:44)
at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource.streamResource(CharSequenceDownloadableResource.java:42)
at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$6.writeTo(ResourceServingHelpers.java:358)
... 115 more
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2128)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2158)
at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:40)
... 117 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1261)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:607)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:597)
at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:581)
at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:272)
at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1510)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:283)
at org.apache.coyote.Response.action(Response.java:173)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:317)
... 125 more

After Some time application came up and available.

2019-12-13 14:48:31,968 http-nio-8080-exec-2001 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/Bm1DWBj3cnAj0R8goVWmpQ00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:51:24,742 http-nio-8080-exec-2113 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/1qKM64mzNIsEEWKrzP6tIw00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:51:46,228 http-nio-8080-exec-2109 INFO [core.event.listener.DirectoryDefaultGroupMembershipResolver] User 'mbient-tech-user' will be added to default groups of directory 'SWF-Default': [artifactory-users, confluence-users, gitlab-users, jenkins-users, jira-software-users]
2019-12-13 14:51:51,910 http-nio-8080-exec-2099 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/WlQvDAfKKGQq7tj1RwzESg00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:53:17,188 http-nio-8080-exec-2044 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/XTgZ0OqQCksdw0NNoUNXCQ00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:54:30,106 http-nio-8080-exec-2111 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/IX0G10E8LJUeseYRPEwUOQ00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:54:37,478 http-nio-8080-exec-2115 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/Pmc4Ch5pbEChx22ifZhsIA00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:54:40,642 http-nio-8080-exec-2109 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/i905O95Zv1cb5sqVAMKT8g00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:58:30,284 http-nio-8080-exec-2040 WARN [common.security.jersey.XsrfResourceFilter] XSRF failure not being enforced for request: https://id.swf.daimler.com/crowd/rest/usermanagement/1/session/cU81dfetY0Y6KjGk0BgdAQ00 , origin: null , referrer: null, method: DELETE
2019-12-13 14:58:30,770 http-nio-8080-exec-2061 INFO [crowd.manager.application.ApplicationServiceGeneric] Invalid credentials for user jasouni in directory SWF-Default (131073), aborting

Kindly check and give update as earliest.