Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,368,302
Community Members
 
Community Events
168
Community Groups

Confluence running on Kubernetes platform crashing with java.lang.OutOfMemoryError: Java heap space

We are running Confluence 7.17.4 Datacenter version on Kubernetes v1.23.4 platform and we are getting OOM errors with very minimal (less that 5 users, none of whom are actively using it daily) usage/users on confluence..

Currently we have the following heap settings 

 resources:
    jvm:
        minHeap: "8092m"
        maxHeap: "8092m" 

and It crashed a few days ago with OOM.

2022-07-30 10:34:55,820 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.LongTaskTimerEvent@58b9c9d5[name=cluster.lock.held.duration,tags={subCategory=current, implementation=com.atlassian.confluence.cluster.hazelcast.HazelcastDualLock, pluginKeyAtCreation=undefined, lockName=scheduler_com.atlassian.confluence.plugins.confluence-notifications-batch-plugin:batchingJob},activeTasks=1,duration=3748.001446] took 6051 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:34:57,727 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@6ef9c432 took 12361 ms (warning threshold is 5000 ms)
2022-07-30 10:35:07,672 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.TimerEvent@76437eda[name=db.ao.entityManager.deleteWithSQL,tags={entityType=com.atlassian.confluence.notifications.batch.ao.NotificationStoreAo, invokerPluginKey=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin},count=1,max=1256.19629,mean=1256.19629,total=1256.19629] took 6176 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:30,528 WARN [alert-dispatch:thread-1] [confluence.alert-log] log 1659177330528 ; WARNING ; JVM ; JVM-1002 ; Garbage collection exceeded time limit ; not-detected ; ; ; {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
[2022-07-30 10:35:30,529] 2022-07-30T10:35:30.528Z Component 'Java' alerted 'Garbage collection exceeded time limit' (details: {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}, trigger: {"pluginKey": "not-detected"})
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:30,529 WARN [alert-dispatch:thread-1] [atlassian-monitor] log 2022-07-30T10:35:30.528Z Component 'Java' alerted 'Garbage collection exceeded time limit' (details: {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}, trigger: {"pluginKey": "not-detected"})
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:31,113 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@2336993e took 5004 ms (warning threshold is 5000 ms)
2022-07-30 10:36:15,015 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.TimerEvent@6149eab9[name=db.sal.transactionalExecutor,tags={taskName=com.atlassian.confluence.extra.calendar3.querydsl.TransactionalQueryDSLSupplier$$Lambda$4550/0x00000008439d7840, invokerPluginKey=com.atlassian.confluence.extra.team-calendars},count=3,max=10527.6407,mean=7728.374765333334,total=23185.124296] took 9868 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:37:45,101 WARN [hz.confluence.migration] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 16" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 18" Exception in thread "I/O dispatcher 14" java.lang.OutOfMemoryError: Java heap space
Exception in thread "scheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,101 WARN [hz.confluence.HealthMonitor] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8090-Poller" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,105 WARN [atlassian-analytics-metrics-publisher] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,756 ERROR [pool-15-thread-7] [springframework.scheduling.support.TaskUtils$LoggingErrorHandler] handleError Unexpected error occurred in scheduled task
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 20" java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:46,399 WARN [HikariPool-1 housekeeper] [zaxxer.hikari.pool.HikariPool] run HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m14s536ms349µs871ns).
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:37:46,400 ERROR [hz.confluence.scheduled.thread-] [com.hazelcast.spi.ExecutionService] log [10.42.7.3]:5701 [confluence] [3.12.11] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@292b1251[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 23" Exception in thread "I/O dispatcher 7" at java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.iterator(ConcurrentHashMap.java:4627)
at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1044)
at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:829)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:30,739 ERROR [pool-15-thread-3] [springframework.scheduling.support.TaskUtils$LoggingErrorHandler] handleError Unexpected error occurred in scheduled task
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:30,739 ERROR [Caesium-1-3] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=JmxAppLoggingJob,deadline=1659176940000]
-- url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:31,417 ERROR [Caesium-1-1] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=JmxLoggingJob,deadline=1659177000000]
-- url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | action: setupcluster
java.lang.OutOfMemoryError: Java heap space

Any suggestions on how to size this without a lot of trial/error? Are there any known issues with this version that would cause this? We are trying to get a profiler set up so we can get more insight into the JVM as well. Seems like 8 gigs should be plenty for it to operate according to the sizing docs we found.

 

Jason

0 answers

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events