Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Confluence going down: ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc

Edited

Greetings, from a few weeks ago, our Confluence environment crashes once or twice a day, and we haven't been able to sort out why.

This is an extract of the log in wich one of the issues took place:

 

2020-02-07 10:02:33,676 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [mchange.v2.resourcepool.BasicResourcePool] log com.mchange.v2.resourcepool.BasicResourcePool@6f5fc0cb -- an attempt to checkout a resource was interrupted, and the pool is still live: some other thread must have interrupted the Thread attempting checkout!
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at net.sf.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:36)
at bucket.core.persistence.hibernate.ConfluenceSharedConnectionProvider.getConnection(ConfluenceSharedConnectionProvider.java:34)
at com.atlassian.hibernate.adapter.adapters.ConnectionProviderV5Adapter.getConnection(ConnectionProviderV5Adapter.java:32)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:505)
at com.atlassian.hibernate.adapter.adapters.session.SessionV2Adapter.connection(SessionV2Adapter.java:160)
at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy34.getTransaction(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy85.getPropertySet(Unknown Source)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserPreferences(DefaultDarkFeaturesManager.java:55)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getUserEnabledFeatures(DefaultDarkFeaturesManager.java:47)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getDarkFeatures(DefaultDarkFeaturesManager.java:99)
at com.atlassian.confluence.setup.settings.DefaultDarkFeaturesManager.getSiteDarkFeatures(DefaultDarkFeaturesManager.java:86)
at sun.reflect.GeneratedMethodAccessor454.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy40.getSiteDarkFeatures(Unknown Source)
at com.atlassian.confluence.mail.notification.listeners.NotificationApiDarkFeature.isEnabled(NotificationApiDarkFeature.java:62)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEventInternal(NotificationsEventDispatcher.java:103)
at com.atlassian.confluence.notifications.impl.NotificationsEventDispatcher.handleEvent(NotificationsEventDispatcher.java:91)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
at com.atlassian.confluence.event.ConfluenceListenerHandlersConfiguration$TimingListenerHandler$1$1.invoke(ConfluenceListenerHandlersConfiguration.java:69)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.lambda$run$0(ConfluenceEventDispatcher.java:93)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:50)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:49)
at com.atlassian.confluence.event.ConfluenceEventDispatcher$VCacheRequestContextRunnableFactory$1.run(ConfluenceEventDispatcher.java:93)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:105)
at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
at com.atlassian.confluence.event.TimingEventPublisher.publish(TimingEventPublisher.java:64)
at com.atlassian.confluence.user.listeners.UserSessionExpiryListener.sessionDestroyed(UserSessionExpiryListener.java:46)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:837)
at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:695)
at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:666)
at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:651)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5680)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:745)
2020-02-07 10:02:33,686 ERROR [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [engine.jdbc.spi.SqlExceptionHelper] logExceptions An SQLException was provoked by the following failure: java.lang.InterruptedException
2020-02-07 10:02:33,693 WARN [ContainerBackgroundProcessor[StandardEngine[Standalone]]] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@38373e8f took 96095 ms (warning threshold is 5000 ms)

 

Also, we have hundres of Broken pipe warnings (maybe they are producing memory leaks?)

and some errors of GC overhead limit exceeded around the same time of the failures.

Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:140)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:198)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:42)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:321)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
at org.apache.coyote.Response.doWrite(Response.java:491)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
... 340 more
2020-02-07 09:59:53,977 WARN [http-nio-8090-exec-33] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888227, threadId=2372, threadName='http-nio-8090-exec-33', userId='xxxxx', type='web-request', summary='/rest/inlinecomments/1.0/comments?containerId=423302281&_=1581065867760'}
-- referer: https://confluence.xxxxxx.com/pages/viewpage.action?pageId=423302281 | url: /rest/inlinecomments/1.0/comments | traceId: 601d20da5498bb64 | userName: xxxxx
2020-02-07 10:00:16,643 WARN [http-nio-8090-exec-56] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@7006d554 took 38149 ms (warning threshold is 5000 ms)
-- url: /login.action | traceId: ac2874dffbb064e0 | userName: anonymous
2020-02-07 10:00:56,903 WARN [http-nio-8090-exec-49] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1581065888197, threadId=2391, threadName='http-nio-8090-exec-49', userId='xxxxx', type='web-request', summary='/plugins/pagetree/naturalchildren.action?decorator=none&excerpt=false&sort=position&reverse=false&disableLinks=false&expandCurrent=true&hasRoot=true&pageId=520946601&treeId=0&startDepth=0&mobile=false&treePageId=549555039&_=1581065880756'}
-- url: /plugins/pagetree/naturalchildren.action | traceId: 7ab747d71365fd89 | userName: xxxxx | referer: https://confluence.xxxxx.com/display/PAR/DC.+Plantilla+plan+de+Medios
2020-02-07 10:00:56,909 ERROR [http-nio-8090-exec-12] [common.error.jersey.ThrowableExceptionMapper] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 8cdc338f5fde9aa0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded
2020-02-07 10:00:56,913 INFO [http-nio-8090-exec-31] [crowd.directory.ldap.SpringLdapTemplateWrapper] call Timed call for search with handler on OU=World,DC=CORP,DC=xxxxx,DC=COM took 55091ms
2020-02-07 10:00:56,918 ERROR [http-nio-8090-exec-13] [common.error.jersey.ThrowableExceptionMapper] toResponse Uncaught exception thrown by REST service: GC overhead limit exceeded
-- referer: https://confluence.xxxxx.com/display/CIR/Grandes+Viajes%3A+Circuitos%2C+Combinados+y+Rutas+en+Coche | url: /rest/com.atlassian.confluence.contributors/1.0/contributors | traceId: 6624b401ad6ac5d0 | userName: xxxxx
java.lang.OutOfMemoryError: GC overhead limit exceeded

 


I searched a bit, but i'm still not versed enough with linux process management to pinpoint the cause.

Any help would be appreciated.

Thank you.

1 answer

Anyone?

we are also facing the same issue. Any suggestion will be appreciated.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Confluence

Confluence Mythbusters: Does Atlassian even use Confluence?

Hi, Confluence collaborators! As part of #Confluence-Collaboratory month, we’ve created a very special Mythsbusters segment, where we're dive into an interesting myth and uncover the truth behind i...

1,761 views 8 32
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you