Always OutOfMemory Exceptions makes Jira Software unstable

[Michael_Klennert]

during and after installation of jira software I became after some minutes or days outOfMemory Exceptions.

My details:

Jira Software Version: 8.8.0

Application Server Container: Apache Tomcat/8.5.42
database: mysql57

OS: Ubuntu 18.04.4 LTS x86_64
Kernel: 4.15.0 
CPU: Intel Xeon E5-2680 v3 (16) @ 1.999GHz
Memory: 32768MiB

I tried following solutions:

in '/opt/atlassian/jira/bin/setenv.sh' increased memory usage to:

JVM_MINIMUM_MEMORY="2048m"
JVM_MAXIMUM_MEMORY="4096m"

in '/etc/security/limits.conf' increased user (jira8 after 8 retries of installations) memory to:

jira8 soft nofile 4096
jira8 hard nofile 8192
jira8 soft nproc 4096
jira8 hard nproc 8192

check with shell:

su jira8

$ ulimit -n
4096

the Exceptions makes working with jira impossible. Please help

[Michael_Klennert · Accepted solution]

Found solution for my problem:

my host provider strato installed the OS in the VM with the following default config:

$> systemctl show --property=DefaultTasksMax

195

Now I increased/changed the property value up to 4192 in:

$> vi /etc/systemd/system.conf

DefaultTasksMax=4192

After restart the OS Jira Software runs stable and doesn't create any Thread problems.

[Moses Thomas]

@Michael_Klennert  You should mark this as accepted solution so that others could use. Glad that you are able to solve  your problem.

Kind regards,

Mo

[Robert Dzido · Accepted solution]

Hi,

Is the machine physical or virtual. What hypervisor are you using if virtual?

[Michael_Klennert]

It's a virtual machine from Strato provider. I think it is Hyper-V from Microsoft or VMware

[Moses Thomas]

@Michael_Klennert  please show logs in <jira_home>/log

[Michael_Klennert]

here is a snapshot from log out of '/var/atlassian/application-data/jira/log/atlassian-jira.log':

...

2020-04-21 13:07:07,966+0200 HealthCheckWatchdog:thread-4 WARN [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Unable to complete execution of health check Threadgrenzwert due to an exception
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.atlassian.troubleshooting.healthcheck.concurrent.SupportHealthCheckTask.getFutureWithTimeout(SupportHealthCheckTask.java:93)
at com.atlassian.troubleshooting.healthcheck.concurrent.SupportHealthCheckTask.lambda$runAsync$0(SupportHealthCheckTask.java:73)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
at java.security.AccessController.doPrivileged(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at com.atlassian.troubleshooting.stp.spi.DefaultRuntimeHelper.spawnProcessSafely(DefaultRuntimeHelper.java:24)
at com.atlassian.troubleshooting.stp.spi.DefaultFileSystemInfo.getThreadLimit(DefaultFileSystemInfo.java:48)
at com.atlassian.troubleshooting.healthcheck.checks.ThreadLimitHealthCheckCondition.shouldDisplay(ThreadLimitHealthCheckCondition.java:32)
at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.shouldDisplay(DefaultSupportHealthCheckSupplier.java:50)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.asPluginSuppliedSupportHealthCheck(DefaultSupportHealthCheckSupplier.java:126)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.lambda$healthChecksFrom$2(DefaultSupportHealthCheckSupplier.java:120)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.healthChecksFrom(DefaultSupportHealthCheckSupplier.java:122)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.byInstance(DefaultSupportHealthCheckSupplier.java:87)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.getHelpPathUrl(SupportHealthStatusBuilder.java:109)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.buildStatus(SupportHealthStatusBuilder.java:134)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.ok(SupportHealthStatusBuilder.java:62)
at com.atlassian.troubleshooting.healthcheck.checks.ThreadLimitHealthCheck.check(ThreadLimitHealthCheck.java:45)
at com.atlassian.troubleshooting.healthcheck.impl.PluginSuppliedSupportHealthCheck.check(PluginSuppliedSupportHealthCheck.java:49)
... 4 more
2020-04-21 13:07:08,001+0200 HealthCheckWatchdog:thread-6 WARN [c.a.t.healthcheck.concurrent.SupportHealthCheckTask] Unable to complete execution of health check Schriftarten due to an exception
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at com.atlassian.troubleshooting.healthcheck.concurrent.SupportHealthCheckTask.getFutureWithTimeout(SupportHealthCheckTask.java:93)
at com.atlassian.troubleshooting.healthcheck.concurrent.SupportHealthCheckTask.lambda$runAsync$0(SupportHealthCheckTask.java:73)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at java.lang.UNIXProcess.initStreams(UNIXProcess.java:288)
at java.security.AccessController.doPrivileged(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:257)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at com.atlassian.troubleshooting.stp.spi.DefaultRuntimeHelper.spawnProcessSafely(DefaultRuntimeHelper.java:24)
at com.atlassian.troubleshooting.stp.spi.DefaultFileSystemInfo.getThreadLimit(DefaultFileSystemInfo.java:48)
at com.atlassian.troubleshooting.healthcheck.checks.ThreadLimitHealthCheckCondition.shouldDisplay(ThreadLimitHealthCheckCondition.java:32)
at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.shouldDisplay(DefaultSupportHealthCheckSupplier.java:50)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.asPluginSuppliedSupportHealthCheck(DefaultSupportHealthCheckSupplier.java:126)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.lambda$healthChecksFrom$2(DefaultSupportHealthCheckSupplier.java:120)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.healthChecksFrom(DefaultSupportHealthCheckSupplier.java:122)
at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.byInstance(DefaultSupportHealthCheckSupplier.java:87)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.getHelpPathUrl(SupportHealthStatusBuilder.java:109)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.buildStatus(SupportHealthStatusBuilder.java:134)
at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.ok(SupportHealthStatusBuilder.java:62)
at com.atlassian.troubleshooting.healthcheck.checks.FontHealthCheck.check(FontHealthCheck.java:29)
at com.atlassian.troubleshooting.healthcheck.impl.PluginSuppliedSupportHealthCheck.check(PluginSuppliedSupportHealthCheck.java:49)
... 4 more

...