Rapid increase in active threads count

[Sergey Shmarkatyuk]

For quite some time my JIRA has been experiencing rapid increase in active threads count. JavaMelody shows it like this: 

jira_active_threads_report.PNG

Usually it goes on and on until I have to manually restart JIRA. At the time of active threads increase some issues hang up and it is not possible to open issue pages of hang up issues. I tried to run garbage collector, it does not help. 

In the past this situation with rapid increase of active threads happened when background indexing was started and when Bulk Clone started. This time neither of those things happen and I would still experience problem with active threads for some reason.

Could please somebody recommend some tweak to resolve this problem? I would appreciate any help.

[Petar Petrov (Appfire)]

Take several thread dumps with several seconds interval. Analyze the stack traces - if there are leaking threads, you'll see many threads with the same stack trace. Looking at the frames, you should be able to pinpoint the faulty add-on or functionality in JIRA that's causing this.

[Sergey Shmarkatyuk]

Hi Petar. Thanks for your answer! One little question I have in mind. Do I need to generate thread dumps when I am experiencing rapid growth of active number of threads or is it ok to generate it when JIRA is running normally?

[Sergey Shmarkatyuk]

I have been able to find threads with the same stack trace:

"StreamsCompletionService::thread-989" #12299 prio=5 os_prio=0 tid=0x0000000025ca7000 nid=0x1d18 waiting on condition [0x000000006188f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005e0f3c0d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
   Locked ownable synchronizers:
	- None

But I don't know how to map it to the faulty plugin functionality as it doesn't give any information about source of the trace. Could you please recommend me how to do this? Thanks.

[Petar Petrov (Appfire)]

You definitely need to take the thread dumps when you see your thread count rising.

That looks like an idling thread from a thread pool. Look for threads with bigger stack traces.

It is normal to see an increase in the number of threads when usage increases - the web server will create more worker threads to handle web requests. I've not used JavaMelody before, but if "active threads" stands for number of threads handling HTTP requests in Tomcat (as their documentation seems to explain) then you should take a tread dump at the peak of usage and look at it. Also check the CPU and memory usage - the reason for high number of active threads could be resource starvation on the server.

[Wojciech Miecznikowski]

Hi Sergey,

Do you have any solution or something what can help find solution?

Regards,

Wojciech Miecznikowski

[Natalya Nekrasova]

Hi

maybe it helps : https://jira.atlassian.com/browse/JSW-12094

[Sergey Shmarkatyuk]

I haven't installed any plugins recently, but I have this problem for quite a while. So, most likely, it is caused by some plugins that were installed long time ago. Any ideas of how I could narrow down the buggy plugin?

[Volodymyr Krupach]

Can be caused by buggy plugins. Have you recently installed any?