Hello,
We recently updated our Jira Server to lateste version.
It was working really good until today, when we inspect catalina.out we found lots of errors like:
'Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread'
Our server is a VPS with 8G RAM memory, Centos 7 with all yum updates done.
As we looked into memory resources we think it shouldn't a problem..
Here follows the full output of /opt/atlassian/jira2/logs/catalina.out:
13-Mar-2019 16:19:53.091 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+PrintGCCause
13-Mar-2019 16:19:53.091 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
13-Mar-2019 16:19:53.091 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/atlassian/jira2
13-Mar-2019 16:19:53.091 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/atlassian/jira2
13-Mar-2019 16:19:53.091 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/atlassian/jira2/temp
13-Mar-2019 16:19:53.092 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
13-Mar-2019 16:19:53.226 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-6969"]
13-Mar-2019 16:19:53.291 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 661 ms
13-Mar-2019 16:19:53.301 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
13-Mar-2019 16:19:53.378 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.35
2019-03-13 16:19:57,103 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/var/atlassian/application-data/jira2' is validated and locked for exclusive use by this instance.
2019-03-13 16:19:57,339 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
****************
JIRA starting...
****************
2019-03-13 16:19:57,457 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
___ Environment _____________________________
JIRA Build : 8.0.1#800009-sha1:f5186e157f3e26c0180673ce49d51cb56d8f7579
Build Date : Wed Feb 20 00:00:00 GMT 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.35 - Servlet API 3.1
Java Version : 1.8.0_181 - Oracle Corporation
Current Working Directory : /opt/atlassian/jira2
Maximum Allowable Memory : 1820MB
Total Memory : 1323MB
Free Memory : 1276MB
Used Memory : 47MB
Memory Pool: Code Cache : Code Cache: init = 33554432(32768K) used = 9795648(9566K) committed = 33554432(32768K) max = 536870912(524288K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 21981456(21466K) committed = 22675456(22144K) max = -1(-1K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 2545536(2485K) committed = 2752512(2688K) max = 1073741824(1048576K)
Memory Pool: PS Eden Space : PS Eden Space: init = 268435456(262144K) used = 27708744(27059K) committed = 626524160(611840K) max = 626524160(611840K)
Memory Pool: PS Survivor Space : PS Survivor Space: init = 44564480(43520K) used = 0(0K) committed = 44564480(43520K) max = 44564480(43520K)
Memory Pool: PS Old Gen : PS Old Gen: init = 716177408(699392K) used = 21130600(20635K) committed = 716177408(699392K) max = 1431830528(1398272K)
JVM Input Arguments : -Djava.util.logging.config.file=/opt/atlassian/jira2/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1024m -Xmx2048m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira2/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira2 -Dcatalina.home=/opt/atlassian/jira2 -Djava.io.tmpdir=/opt/atlassian/jira2/temp
Java Compatibility Information : JIRA version = 8.0.1, Java Version = 1.8.0_181
___ Java System Properties _________________
atlassian.plugins.startup.options :
atlassian.standalone : JIRA
awt.toolkit : sun.awt.X11.XToolkit
catalina.base : /opt/atlassian/jira2
catalina.home : /opt/atlassian/jira2
catalina.useNaming : true
common.loader : "${catalina.base}/lib",
"${catalina.base}/lib/*.jar",
"${catalina.home}/lib",
"${catalina.home}/lib/*.jar"
file.encoding : ANSI_X3.4-1968
file.encoding.pkg : sun.io
ignore.endorsed.dirs :
java.awt.graphicsenv : sun.awt.X11GraphicsEnvironment
java.awt.headless : true
java.awt.printerjob : sun.print.PSPrinterJob
java.class.version : 52.0
java.home : /opt/atlassian/jira2/jre
java.io.tmpdir : /opt/atlassian/jira2/temp
java.locale.providers : COMPAT
java.naming.factory.initial : org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs : org.apache.naming
java.protocol.handler.pkgs : org.apache.catalina.webresources
java.runtime.name : Java(TM) SE Runtime Environment
java.runtime.version : 1.8.0_181-b13
java.specification.name : Java Platform API Specification
java.specification.vendor : Oracle Corporation
java.specification.version : 1.8
java.util.concurrent.ForkJoinPool.common.threadFactory : org.apache.catalina.startup.SafeForkJoinWorkerThreadFactory
java.util.logging.config.file : /opt/atlassian/jira2/conf/logging.properties
java.util.logging.manager : org.apache.juli.ClassLoaderLogManager
java.vendor : Oracle Corporation
java.vendor.url : http://java.oracle.com/
java.vendor.url.bug : http://bugreport.sun.com/bugreport/
java.version : 1.8.0_181
java.vm.info : mixed mode
java.vm.name : Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name : Java Virtual Machine Specification
java.vm.specification.vendor : Oracle Corporation
java.vm.specification.version : 1.8
java.vm.vendor : Oracle Corporation
java.vm.version : 25.181-b13
jdk.tls.ephemeralDHKeySize : 2048
mail.mime.decodeparameters : true
org.apache.catalina.security.SecurityListener.UMASK : 0027
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER : true
org.dom4j.factory : com.atlassian.core.xml.InterningDocumentFactory
os.arch : amd64
os.name : Linux
os.version : 2.6.32-042stab127.2
package.access : sun.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.tomcat.
package.definition : sun.,
java.,
org.apache.catalina.,
org.apache.coyote.,
org.apache.jasper.,
org.apache.naming.,
org.apache.tomcat.
server.loader :
shared.loader :
sun.arch.data.model : 64
sun.boot.library.path : /opt/atlassian/jira2/jre/lib/amd64
sun.cpu.endian : little
sun.cpu.isalist :
sun.io.unicode.encoding : UnicodeLittle
sun.java.command : org.apache.catalina.startup.Bootstrap start
sun.java.launcher : SUN_STANDARD
sun.jnu.encoding : ANSI_X3.4-1968
sun.management.compiler : HotSpot 64-Bit Tiered Compilers
sun.os.patch.level : unknown
tomcat.util.buf.StringCache.byte.enabled : true
tomcat.util.scan.StandardJarScanFilter.jarsToScan : log4j-taglib*.jar,
log4j-web*.jar,
log4javascript*.jar,
slf4j-taglib*.jar
tomcat.util.scan.StandardJarScanFilter.jarsToSkip : annotations-api.jar,
ant-junit*.jar,
ant-launcher.jar,
ant.jar,
asm-*.jar,
aspectj*.jar,
bootstrap.jar,
catalina-ant.jar,
catalina-ha.jar,
catalina-jmx-remote.jar,
catalina-storeconfig.jar,
catalina-tribes.jar,
catalina-ws.jar,
catalina.jar,
cglib-*.jar,
cobertura-*.jar,
commons-beanutils*.jar,
commons-codec*.jar,
commons-collections*.jar,
commons-daemon.jar,
commons-dbcp*.jar,
commons-digester*.jar,
commons-fileupload*.jar,
commons-httpclient*.jar,
commons-io*.jar,
commons-lang*.jar,
commons-logging*.jar,
commons-math*.jar,
commons-pool*.jar,
dom4j-*.jar,
easymock-*.jar,
ecj-*.jar,
el-api.jar,
geronimo-spec-jaxrpc*.jar,
h2*.jar,
hamcrest-*.jar,
hibernate*.jar,
httpclient*.jar,
icu4j-*.jar,
jasper-el.jar,
jasper.jar,
jaspic-api.jar,
jaxb-*.jar,
jaxen-*.jar,
jdom-*.jar,
jetty-*.jar,
jmx-tools.jar,
jmx.jar,
jsp-api.jar,
jstl.jar,
jta*.jar,
junit-*.jar,
junit.jar,
log4j*.jar,
mail*.jar,
objenesis-*.jar,
oraclepki.jar,
oro-*.jar,
servlet-api-*.jar,
servlet-api.jar,
slf4j*.jar,
taglibs-standard-spec-*.jar,
tagsoup-*.jar,
tomcat-api.jar,
tomcat-coyote.jar,
tomcat-dbcp.jar,
tomcat-i18n-en.jar,
tomcat-i18n-es.jar,
tomcat-i18n-fr.jar,
tomcat-i18n-ja.jar,
tomcat-i18n-ru.jar,
tomcat-jdbc.jar,
tomcat-jni.jar,
tomcat-juli-adapters.jar,
tomcat-juli.jar,
tomcat-util-scan.jar,
tomcat-util.jar,
tomcat-websocket.jar,
tools.jar,
websocket-api.jar,
wsdl4j*.jar,
xercesImpl.jar,
xml-apis.jar,
xmlParserAPIs-*.jar,
xmlParserAPIs.jar,
xom-*.jar
user.country : US
user.dir : /opt/atlassian/jira2
user.home : /home/jira
user.language : en
user.name : jira
user.timezone : GMT
2019-03-13 16:19:57,628 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira2/dbconfig.xml
2019-03-13 16:19:57,918 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] Running JIRA startup checks.
2019-03-13 16:19:58,009 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] JIRA pre-database startup checks completed successfully.
2019-03-13 16:19:58,189 JIRA-Bootstrap INFO [c.a.j.config.database.SystemDatabaseConfigurationLoader] Reading database configuration from /var/atlassian/application-data/jira2/dbconfig.xml
13-Mar-2019 16:19:58.445 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-6969"]
13-Mar-2019 16:19:58.605 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
We would appreciate some help solving this problem.
Best Regards,
Pedro Marques
