Rapid increase in active threads count

For quite some time my JIRA has been experiencing rapid increase in active threads count. JavaMelody shows it like this: 

jira_active_threads_report.PNG

Usually it goes on and on until I have to manually restart JIRA. At the time of active threads increase some issues hang up and it is not possible to open issue pages of hang up issues. I tried to run garbage collector, it does not help. 

In the past this situation with rapid increase of active threads happened when background indexing was started and when Bulk Clone started. This time neither of those things happen and I would still experience problem with active threads for some reason.

Could please somebody recommend some tweak to resolve this problem? I would appreciate any help.

4 answers

Take several thread dumps with several seconds interval. Analyze the stack traces - if there are leaking threads, you'll see many threads with the same stack trace. Looking at the frames, you should be able to pinpoint the faulty add-on or functionality in JIRA that's causing this.

Hi Petar. Thanks for your answer! One little question I have in mind. Do I need to generate thread dumps when I am experiencing rapid growth of active number of threads or is it ok to generate it when JIRA is running normally?

I have been able to find threads with the same stack trace:

"StreamsCompletionService::thread-989" #12299 prio=5 os_prio=0 tid=0x0000000025ca7000 nid=0x1d18 waiting on condition [0x000000006188f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005e0f3c0d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
	at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
   Locked ownable synchronizers:
	- None

But I don't know how to map it to the faulty plugin functionality as it doesn't give any information about source of the trace. Could you please recommend me how to do this? Thanks.

You definitely need to take the thread dumps when you see your thread count rising.

That looks like an idling thread from a thread pool. Look for threads with bigger stack traces.

It is normal to see an increase in the number of threads when usage increases - the web server will create more worker threads to handle web requests. I've not used JavaMelody before, but if "active threads" stands for number of threads handling HTTP requests in Tomcat (as their documentation seems to explain) then you should take a tread dump at the peak of usage and look at it. Also check the CPU and memory usage - the reason for high number of active threads could be resource starvation on the server.

Can be caused by buggy plugins. Have you recently installed any?

I haven't installed any plugins recently, but I have this problem for quite a while. So, most likely, it is caused by some plugins that were installed long time ago. Any ideas of how I could narrow down the buggy plugin?

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published yesterday in Hipchat

Moving from Hipchat to Stride? Here’s what you’ll love

Heya, Hipchat friends! We’re so happy you’re checking out   Stride. Whether you know it or not, you have been instrumental in making Stride come to life. Every feature, design, and functionality...

68 views 2 6
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you