High CPU, lot of long running threads on lucene indexes in JIRA 6.0.8

Hi,

I have a situation where my server CPU utilization starts increasing and stays on for a very long time (haven't measured exactly how long, i have see it high for about 1-2 hrs)

JIRA becomes unresponsive and everything seems to fall back to normal upon restart

I took thread dumps(20 every 3 seconds ) during the hign CPU time and i get the below stack for all the long running threads

I am not able to figureout, why high CPU utilization stays on for a very long time and comes back to normal after restart.

Most of these long running threads originate from , JIRA Ajile, gadgets, Script Runner other JQLs

at sun.misc.Unsafe.getByte(Native Method) at java.nio.DirectByteBuffer.get(Unknown Source) at java.nio.ByteBuffer.get(Unknown Source) at java.nio.DirectByteBuffer.get(Unknown Source) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.readBytes(MMapDirectory.java:289) at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) at com.atlassian.jira.issue.search.parameters.lucene.sort.JiraLuceneFieldFinder.getMatches(JiraLuceneFieldFinder.java:232) at com.atlassian.jira.issue.search.parameters.lucene.sort.JiraLuceneFieldFinder.getUniqueMatches(JiraLuceneFieldFinder.java:184) at com.atlassian.jira.issue.statistics.util.OneDimensionalObjectHitCollector.setNextReader(OneDimensionalObjectHitCollector.java:92) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:531) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:320) at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:153)

thanks in advanace.

2 answers

Hi Sridhar,

you can try to rebuild your indexes from ground. You'll find the indexes in this folder \DATA\caches\indexes\.

1. Shutdown JIRA

2. Delete indexes

3. Start JIRA

4. Go to administration panel and recreate indexes

I hope this will help you!

That didn't work..

I have almost 90 threads with the below stack(cut off the remaining stack), which is using up most (> 80%)of the CPU, all in runnable state

at org.apache.lucene.store.DataInput.readVInt(DataInput.java:105)
at org.apache.lucene.index.SegmentTermDocs.read(SegmentTermDocs.java:144)
at org.apache.lucene.search.TermScorer.nextDoc(TermScorer.java:112)
at org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:127)
at org.apache.lucene.search.BooleanScorer2.nextDoc(BooleanScorer2.java:303)
at org.apache.lucene.util.OpenBitSetDISI.inPlaceOr(OpenBitSetDISI.java:52)
at org.apache.lucene.util.OpenBitSetDISI.(OpenBitSetDISI.java:32)
at org.apache.lucene.search.CachingWrapperFilter.docIdSetToCache(CachingWrapperFilter.java:181)
at org.apache.lucene.search.CachingWrapperFilter.getDocIdSet(CachingWrapperFilter.java:203)
at org.apache.lucene.search.IndexSearcher.searchWithFilter(IndexSearcher.java:551)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:532)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:433)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:356)

We are facing same issue. Any fixes available to try?

Can you please elobarate the issue you are facing? do you have the same thread pile up of lucene indexes ?

We have a powerful server and in spite of that almost every day, the CPU load increases and JIRA becomes unresponsive. Therad dumps reveal that threads piling up on Lucene search. We are on JIRA 6.0.6. I see that you faced this problem back in May. Are you still facing this? If not, did you try any fixes?

Can you compare the thread dump when there is high CPU and when the CPU is OK and post the threads that are piling up

Suggest an answer

Log in or Join to answer
Community showcase
Sarah Schuster
Posted Jan 29, 2018 in Jira

What are common themes you've seen across successful & failed Jira Software implementations?

Hey everyone! My name is Sarah Schuster, and I'm a Customer Success Manager in Atlassian specializing in Jira Software Cloud. Over the next few weeks I will be posting discussion topics (8 total) to ...

2,811 views 11 18
Join discussion

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you
Atlassian Team Tour

Join us on the Team Tour

We're bringing product updates and pro tips on teamwork to ten cities around the world.

Save your spot