Monitoring total jira memory usage

[Derek Murphy]

Hi,

I'm trying to come up with a way to get very specific stats from the JVM running jira. I would like to be able to query whatever is generating the below stats, so I know if my memory usage is ever above 80%. Even being able to get the total used would be useful, but I can't seem to find a tool that will simply give me that. We had some out of memory errors before I turned my Xms to 1gb and Xmx to 2gb, but I'd like to proactively be able to find out if I'm getting close to out of memory errors, hopefully using something like jstat.

Does anyone have any tips? I'm not that familiar with using these tools.

[jira@jiraserver bin]$ ./jstat -gcutil -t 30317 5000
Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT
906322.1 48.71 0.00 92.70 60.32 99.94 132 1.906 0 0.000 1.906
906327.1 48.71 0.00 92.70 60.32 99.94 132 1.906 0 0.000 1.906
906332.1 48.71 0.00 92.71 60.32 99.94 132 1.906 0 0.000 1.906
906337.1 48.71 0.00 92.71 60.32 99.94 132 1.906 0 0.000 1.906
906342.1 48.71 0.00 94.92 60.32 99.94 132 1.906 0 0.000 1.906
906347.1 48.71 0.00 98.32 60.32 99.94 132 1.906 0 0.000 1.906
906352.1 48.71 0.00 98.32 60.32 99.94 132 1.906 0 0.000 1.906
906357.1 48.71 0.00 99.73 60.32 99.94 132 1.906 0 0.000 1.906
906362.1 0.00 48.18 3.65 60.52 99.94 133 1.918 0 0.000 1.918
906367.1 0.00 48.18 3.73 60.52 99.94 133 1.918 0 0.000 1.918
906372.1 0.00 48.18 3.75 60.52 99.94 133 1.918 0 0.000 1.918
906377.1 0.00 48.18 3.75 60.52 99.94 133 1.918 0 0.000 1.918
906382.1 0.00 48.18 3.76 60.52 99.94 133 1.918 0 0.000 1.918
906387.1 0.00 48.18 8.89 60.52 99.94 133 1.918 0 0.000 1.918
906392.1 0.00 48.18 27.08 60.52 99.95 133 1.918 0 0.000 1.918
906397.1 0.00 48.18 30.34 60.52 99.95 133 1.918 0 0.000 1.918
906402.1 0.00 48.18 35.27 60.52 99.95 133 1.918 0 0.000 1.918
906407.1 0.00 48.18 45.83 60.52 99.96 133 1.918 0 0.000 1.918

Java VM Memory Statistics
Memory Graph	76 % Free (Used: 435 MB Total: 1820MB) (Force garbage collection)

[MatthewC]

Have you thought about collecting the GC logs? Add these commands to your Jira startup (tomcat6w //ES//Jira)

-XX:+PrintGCTimeStamps -verbose:gc -Xloggc:<absolute path>/atlassian-gc.log -XX:+HeapDumpOnOutOfMemoryError

We also have JMX enabled & use VisualVM and jconsole to keep an eye on Jira & see how the different parts of the memory are filling up. To enable it you need to pass these parameters to the service:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

For more examples see: http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html#Enabling%20JMX%20Remote

References:

http://confluence.atlassian.com/display/JIRAKB/Garbage+Collection+Performance+Problems

http://java.sun.com/developer/technicalArticles/Programming/GCPortal/