JIRA - full GC is run increasingly often

Andrei [errno]

... this might be a 1st Q of many to follow... trying to get to the root of the slow performing JIRA instance.

so: after a few 60+ sec delays we started monitoring GC logs (feeding from gc.log), looking for the likes of:

[Full GC (Metadata GC Threshold) [PSYoungGen: 34739K->0K(1223168K)] [ParOldGen: 184K->33658K(2796544K)] 34923K->33658K(4019712K), [Metaspace: 20904K->20904K(1069056K)], 0.0975657 secs] [Times: user=0.45 sys=0.04, real=0.10 secs]
...
[Full GC (System.gc()) [PSYoungGen: 6168K->0K(2101760K)] [ParOldGen: 33666K->38319K(2796544K)] 39835K->38319K(4898304K), [Metaspace: 25441K->25441K(1073152K)], 0.1013057 secs] [Times: user=0.56 sys=0.01, real=0.11 secs]
...
[Full GC (Ergonomics) [PSYoungGen: 2754374K->0K(2755584K)] [ParOldGen: 5592570K->3039221K(5592576K)] 8346945K->3039221K(8348160K), [Metaspace: 402263K->396816K(1433600K)], 13.2142183 secs] [Times: user=83.80 sys=2.88, real=13.21 secs]

I took a look at pstree and noticed what I think is a high number of java threads... is it normal?
the box runs only JIRA/with bundled Tomcat + nginx + postgres, no other java stuff...

jira ~ pstree
init─┬─atd
     ├─crond
     ├─dhclient
     ├─java───5767*[{java}]
     ├─6*[mingetty]
     ├─nginx───2*[nginx]
     ├─postmaster───16*[postmaster]
...skip

thanks!

crf

If 5767 is the number of threads, then absolutely not.  Depending on how busy the system is and how you have it configured, I wouldn't normally expect to see more than a few hundred or so.  Whatever these threads are doing, it seems likely that they are related to your GC problems.

I would suggest using jstack to dump the threads.  The output format can vary slightly by operating system, but this is likely to give you something interesting:

jstack [JIRA_Process_ID_Here] | grep tid= | cut -d\" -f2 | sort | uniq -c

 

The output should look something like this:

cfuller@crf:~$ jstack 40568 | grep tid= | cut -d\" -f2 | sort | uniq -c
   1 Attach Listener
   1 C1 CompilerThread3
   1 C2 CompilerThread0
   1 C2 CompilerThread1
   1 C2 CompilerThread2
   1 DestroyJavaVM
   1 Finalizer
   1 GC task thread#0 (ParallelGC)
...

 

This is the count of threads with a given name, so if there are many duplicates of a single kind of thread, it should show up.  You should also be able to see if there is a whole series of several threads with similar names, like "pool-8-thread-1" through "pool-8-thread-5601".  In this latter case, since the thread name isn't helpful, jstack would have to catch one of these naughty threads in the act of doing something interesting for you to figure out where it came from.