JIRA GC Problems

N_o0x48p

Hi everyone!

We have big problems with prom server JIRA 7.2.2.
Every working day (everything is good in weekends) unsuddenly CPU is overloaded (>10 CORE) one time a day, users can't load pages. Log files stop log. Before it does not contain any specific errors. JAVA Melody usually is not available at this moment.

After reboot, JAVA Melody GC Collector Chart shows that time of processing garbage became extra large before reboot (in other time chart is empty as must be).
I viewed GC logs via GC viewer and didn't discover any new information.

Can you recommend something? Some other instruments for diagnostic instead of GC Viewer and Java Melody?  We have increased memory a few times. Nothing changed.

 java arguments:

-Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms8192m -Xmx9216m -Dfile.encoding=utf-8 -Dsun.jnu.encoding=UTF-8 -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Djavax.net.ssl.trustStore=/etc/pki/ca-trust/extracted/java/cacerts -Dupm.pac.disable=true -Datlassian.upm.server.data.disable=true -XX:-OmitStackTraceInFastThrow -XX:ReservedCodeCacheSize=512m -XX:+UseCodeCacheFlushing -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics -XX:+PrintJNIGCStalls -XX:+DisableExplicitGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -Xloggc:/opt/atlassian/application-data/jira/log/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Dcom.sun.management.jmxremote.password.file=/opt/atlassian/jira/conf/remote.users -Dcom.sun.management.jmxremote.access.file=/opt/atlassian/jira/conf/remote.acl -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3312 -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=srv-jira.bft.local -Djava.endorsed.dirs=/opt/atlassian/jira/endorsed -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp

 

Daniel Eads

Thanks for including your java arguments! The behavior you described is pretty consistent with a stop-the-world garbage collection event.

Switching from the CMS collector to G1GC would prevent the application from stalling during a full garbage collection like you're seeing now. It does come with its own performance tradeoffs, I'd recommend testing that in a staging environment if you've got a clone of your production instance to make sure things come up nicely before switching your production instance over. To enable G1GC, use this flag:

-XX:+UseG1GC 

You'll also need to remove the CMS flags currently enabled:

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC

 

We've got a nice discussion in a previous thread about some resources for performance tuning (the resources are still valuable for Jira despite the thread being about Confluence). If you've got the time to work on this, I'd recommend reading through all the resources linked in that thread. Garbage Collection tuning can be a bit of a dark art, so it's important to take your time to avoid doing more harm than good. With the logging and analysis you've done so far, I think you're headed in the right direction!

Cheers,
Daniel