Confluence going down: ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc

[ISS Service]

Greetings, from a few weeks ago, our Confluence environment crashes once or twice a day, and we haven't been able to sort out why.

This is an extract of the log in wich one of the issues took place:

2020-02-07 10:02:33,676 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool@6f5fc0cb -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:36)
at bucket.core.persistence.hibernate.ConfluenceSharedConnectionProvider.getConnection(ConfluenceSharedConnectionProvider.java:34)
at com.atlassian.hibernate.adapter.adapters.ConnectionProviderV5Adapter.getConnection(ConnectionProviderV5Adapter.java:32)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:505)
at com.atlassian.hibernate.adapter.adapters.session.SessionV2Adapter.connection(SessionV2Adapter.java:160)
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy34.getTransaction(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy85.getPropertySet(Unknown Source)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserPreferences(DefaultDarkFeaturesManager.java:55)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserEnabledFeatures(DefaultDarkFeaturesManager.java:47)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getDarkFeatures(DefaultDarkFeaturesManager.java:99)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getSiteDarkFeatures(DefaultDarkFeaturesManager.java:86)
at sun.reflect.GeneratedMethodAccessor454.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy40.getSiteDarkFeatures(Unknown Source)
at com.atlassian.confluence.mail.notification.listeners.NotificationApiDarkFeature.isEnabled(NotificationApiDarkFeature.java:62)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEventInternal(NotificationsEventDispatcher.java:103)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEvent(NotificationsEventDispatcher.java:91)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
at com.atlassian.confluence.event.ConfluenceListenerHandlersConfiguration$TimingListenerHandler$1$1.invoke(ConfluenceListenerHandlersConfiguration.java:69)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.lambda$run$0(ConfluenceEventDispatcher.java:93)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:50)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:49)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.run(ConfluenceEventDispatcher.java:93)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:105)
at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:64)
at com.atlassian.confluence.user.listeners.UserSessionExpiryListener.sessionDestroyed(UserSessionExpiryListener.java:46)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:837)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:695)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:666)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:651)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5680)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:745)
2020-02-07 10:02:33,686 ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc.spi.SqlExceptionHelper] logExceptions An SQLException was provoked by the following failure: java.lang.InterruptedException
2020-02-07 10:02:33,693 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@38373e8f took 96095 ms (warning threshold is 5000 ms)

Also, we have hundres of Broken pipe warnings (maybe they are producing memory leaks?)

and some errors of GC overhead limit exceeded around the same time of the failures.

Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:140)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:198)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:42)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:321)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
at org.apache.coyote.Response.doWrite(Response.java:491)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
... 340 more
2020-02-07 09:59:53,977 WARN [http-nio-8090-exec-33] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888227, threadId=2372, threadName='http-nio-8090-exec-33', userId='xxxxx', type='web-request', summary='/rest/inlinecomments/1.0/comments?containerId=423302281&_=1581065867760'}
-- referer: https://confluence.xxxxxx.com/pages/viewpage.action?pageId=423302281 | url: /rest/inlinecomments/1.0/comments | traceId: 601d20da5498bb64 | userName: xxxxx
2020-02-07 10:00:16,643 WARN [http-nio-8090-exec-56] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@7006d554 took 38149 ms (warning threshold is 5000 ms)
-- url: /login.action | traceId: ac2874dffbb064e0 | userName: anonymous
2020-02-07 10:00:56,903 WARN [http-nio-8090-exec-49] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888197, threadId=2391, threadName='http-nio-8090-exec-49', userId='xxxxx', type='web-request', summary='/plugins/pagetree/naturalchildren.action?decorator=none&excerpt=false&sort=position&reverse=false&disableLinks=false&expandCurrent=true&hasRoot=true&pageId=520946601&treeId=0&startDepth=0&mobile=false&treePageId=549555039&_=1581065880756'}
-- url: /plugins/pagetree/naturalchildren.action | traceId: 7ab747d71365fd89 | userName: xxxxx | referer: https://confluence.xxxxx.com/display/PAR/DC.+Plantilla+plan+de+Medios
2020-02-07 10:00:56,909 ERROR [http-nio-8090-exec-12] [common.error.jersey.ThrowableExceptionMapper] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 8cdc338f5fde9aa0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-02-07 10:00:56,913 INFO [http-nio-8090-exec-31] [crowd.directory.ldap.SpringLdapTemplateWrapper] call Timed call for search with handler on OU=World,DC=CORP,DC=xxxxx,DC=COM took 55091ms
2020-02-07 10:00:56,918 ERROR [http-nio-8090-exec-13] [common.error.jersey.ThrowableExceptionMapper] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 6624b401ad6ac5d0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded

I searched a bit, but i'm still not versed enough with linux process management to pinpoint the cause.

Any help would be appreciated.

Thank you.

[ISS Service]

Anyone?

[papireddy]

we are also facing the same issue. Any suggestion will be appreciated.