Investigate Crash site with logs, ...catalina.connector.ClientAbortException

[Vinh Le]

Hi ,

Is there anyone out there could help decipher this logs bc the last 6 months our Confluence has crashed almost every 2 days. It was running fine for a year or so.

I wonder if this log can give me any clue where or how to fix this.

We are using AWS instance t2 Large , 8G ram and 2 vcpu , should be fine.

It is not a big site too.

Confluence server version 6.12.4

below is an actually sample of the exception logs:

edit:

This is the unhandled exception which is very similar to one of the post here:

2019-11-19 05:30:08,443 ERROR [Caesium-1-4] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Connection reset by peer: socket write error
2019-11-19 05:30:08,443 ERROR [Caesium-1-4] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=flushEdgeIndexQueue,deadline=1574101808418]
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:473)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy53.getTransaction(Unknown Source)

-----

......

-- url: /rest/scroll-pdf/1.0/sync-export | traceId: 01aaeb10d1a097d7 | userName: anonymous
2019-11-20 06:15:17,456 ERROR [https-jsse-nio2-443-exec-24] [confluence.web.rest.LegacySynchronousExportResource] write Error while streaming data to the client.
-- url: /rest/scroll-pdf/1.0/sync-export | traceId: 01aaeb10d1a097d7 | userName: anonymous
org.apache.catalina.connector.ClientAbortException: java.io.IOException: The specified network name is no longer available.

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:333)
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:758)
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:663)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:368)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:346)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
at com.atlassian.confluence.web.filter.ResponseOutputStreamFilter$NoopAfterCloseOutputStream.write(ResponseOutputStreamFilter.java:100)
at com.atlassian.gzipfilter.RoutableServletOutputStream.write(RoutableServletOutputStream.java:134)
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
at com.k15t.scroll.confluence.web.rest.SynchronousExportResource$4$1.write(SynchronousExportResource.java:245)
at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:78)
at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:60)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at com.k15t.scroll.exporter.infra.AbstractJerseyServletContainer.service(AbstractJerseyServletContainer.java:67)
at com.atlassian.plugin.servlet.DelegatingPluginServlet.service(DelegatingPluginServlet.java:37)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at com.atlassian.plugin.servlet.ServletModuleContainerServlet.service(ServletModuleContainerServlet.java:45)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
at com.atlassian.confluence.plugins.baseurl.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:52)
at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)

[Shannon S]

Hello Vinh,

Thank you for contacting us about this.

Could you tell me, is this a Confluence Data Center installation, since you're on AWS? Where is your database being hosted, and what type of database is it?

The error looks like it can't connect to your database for some reason. Are you running behind any firewall or proxy that could prevent that?

I would also recommend taking a look at the suggestions on a thread that I worked on that had a similar error: Why was my Confluence-instance down? 

Regards,

Shannon

[Vinh Le]

Appreciate you help as i am desperate. It is a live site and our customers are furious.

It is a VM hosted by AWS. Insdie the VM, we install Confluence server. This VM also hosts of the MS SQL express server.

It has been running without any crashed for a year or so but start this year also crashes every day.

[Shannon S]

Vinh,

No worries, I'm here for you! 

I would recommend that you first have a look at Confluence slows and times out during periods of high load due to database connection pool. Your message is related to the configuration of the JDBC pool or to performance issues on the database, and the above article can help you. 

If that doesn't work, then I would recommend that you migrate to a supported database version. MS SQL Express is not supported. You can find a list of supported databases from our documentation. See migrating to another database for information on how to change databases.

Keep me updated!

Regards,

Shannon