Confluence running on Kubernetes platform crashing with java.lang.OutOfMemoryError: Java heap space

[Jason Welsh]

We are running Confluence 7.17.4 Datacenter version on Kubernetes v1.23.4 platform and we are getting OOM errors with very minimal (less that 5 users, none of whom are actively using it daily) usage/users on confluence..

Currently we have the following heap settings 

 resources:
    jvm:
        minHeap: "8092m"
        maxHeap: "8092m" 

and It crashed a few days ago with OOM.

2022-07-30 10:34:55,820 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.LongTaskTimerEvent@58b9c9d5[name=cluster.lock.held.duration,tags={subCategory=current, implementation=com.atlassian.confluence.cluster.hazelcast.HazelcastDualLock, pluginKeyAtCreation=undefined, lockName=scheduler_com.atlassian.confluence.plugins.confluence-notifications-batch-plugin:batchingJob},activeTasks=1,duration=3748.001446] took 6051 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:34:57,727 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@6ef9c432 took 12361 ms (warning threshold is 5000 ms)
2022-07-30 10:35:07,672 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.TimerEvent@76437eda[name=db.ao.entityManager.deleteWithSQL,tags={entityType=com.atlassian.confluence.notifications.batch.ao.NotificationStoreAo, invokerPluginKey=com.atlassian.confluence.plugins.confluence-notifications-batch-plugin},count=1,max=1256.19629,mean=1256.19629,total=1256.19629] took 6176 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:30,528 WARN [alert-dispatch:thread-1] [confluence.alert-log] log 1659177330528 ; WARNING ; JVM ; JVM-1002 ; Garbage collection exceeded time limit ; not-detected ; ; ; {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
[2022-07-30 10:35:30,529] 2022-07-30T10:35:30.528Z Component 'Java' alerted 'Garbage collection exceeded time limit' (details: {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}, trigger: {"pluginKey": "not-detected"})
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:30,529 WARN [alert-dispatch:thread-1] [atlassian-monitor] log 2022-07-30T10:35:30.528Z Component 'Java' alerted 'Garbage collection exceeded time limit' (details: {"durationInMillis":8785,"windowInMillis":20000,"limitPercent":10,"threadMemoryAllocations":"","threadDump":[]}, trigger: {"pluginKey": "not-detected"})
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:35:31,113 WARN [Catalina-utility-1] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.sal.api.events.SessionDestroyedEvent@2336993e took 5004 ms (warning threshold is 5000 ms)
2022-07-30 10:36:15,015 WARN [atlassian-analytics-metrics-publisher] [confluence.util.profiling.DurationThresholdWarningTimingHelperFactory] logMessage Execution time for publishing event com.atlassian.util.profiling.micrometer.analytics.events.TimerEvent@6149eab9[name=db.sal.transactionalExecutor,tags={taskName=com.atlassian.confluence.extra.calendar3.querydsl.TransactionalQueryDSLSupplier$$Lambda$4550/0x00000008439d7840, invokerPluginKey=com.atlassian.confluence.extra.team-calendars},count=3,max=10527.6407,mean=7728.374765333334,total=23185.124296] took 9868 ms (warning threshold is 5000 ms)
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:37:45,101 WARN [hz.confluence.migration] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 16" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 18" Exception in thread "I/O dispatcher 14" java.lang.OutOfMemoryError: Java heap space
Exception in thread "scheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,101 WARN [hz.confluence.HealthMonitor] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8090-Poller" java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 8" java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,105 WARN [atlassian-analytics-metrics-publisher] [confluence.cluster.hazelcast.ConfluenceOutOfMemoryHandler] onOutOfMemory OutOfMemoryError occurred attempting to continue operating as normal
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:45,756 ERROR [pool-15-thread-7] [springframework.scheduling.support.TaskUtils$LoggingErrorHandler] handleError Unexpected error occurred in scheduled task
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 20" java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:37:46,399 WARN [HikariPool-1 housekeeper] [zaxxer.hikari.pool.HikariPool] run HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m14s536ms349µs871ns).
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
2022-07-30 10:37:46,400 ERROR [hz.confluence.scheduled.thread-] [com.hazelcast.spi.ExecutionService] log [10.42.7.3]:5701 [confluence] [3.12.11] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@292b1251[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
Exception in thread "I/O dispatcher 23" Exception in thread "I/O dispatcher 7" at java.base/java.util.concurrent.ConcurrentHashMap$KeySetView.iterator(ConcurrentHashMap.java:4627)
at java.base/java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1044)
at java.base/java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:210)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:829)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:30,739 ERROR [pool-15-thread-3] [springframework.scheduling.support.TaskUtils$LoggingErrorHandler] handleError Unexpected error occurred in scheduled task
-- referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | url: /setup/setupdbtype.action | traceId: 00115abd9696f6e1 | userName: anonymous | action: setupdbtype
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:30,739 ERROR [Caesium-1-3] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=JmxAppLoggingJob,deadline=1659176940000]
-- url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | action: setupcluster
java.lang.OutOfMemoryError: Java heap space
2022-07-30 10:38:31,417 ERROR [Caesium-1-1] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=JmxLoggingJob,deadline=1659177000000]
-- url: /setup/setupcluster.action | traceId: 0070dcceec079644 | userName: anonymous | referer: https://confluence.10-21-34-11.nowhere.io/setup/setuplicense.action | action: setupcluster
java.lang.OutOfMemoryError: Java heap space

Any suggestions on how to size this without a lot of trial/error? Are there any known issues with this version that would cause this? We are trying to get a profiler set up so we can get more insight into the JVM as well. Seems like 8 gigs should be plenty for it to operate according to the sizing docs we found.

Jason