Increase reindex speed - jira.index.issue.threads not showing in Advanced Settings

[Sha Aquino]

Hi,

We are trying to increase our reindex speed because it is currently at the rate of 1% in 1hour.

I referred to this page: https://confluence.atlassian.com/jirakb/how-to-increase-the-speed-of-full-reindex-in-jira-826879636.html

Our current version: Jira Software 7.13.0

I have verified our home directory via Server info and jira-application.properties.

I created the jira-config.properties fiel in our home directory.

First time value:

jira.index.issue.threads = 20

 Result: No entry in Advanced Settings

Second time value:

# jira.index.issue.threads = 10
jira.index.issue.threads = 20

Result: Still no entry in Advanced Settings.

Was hoping maybe anyone could guide me to the right path? We seem to be stuck.

Thanks in advance for any help. :)

[Ananias Gutierrez]

Hello @Nic Brough -Adaptavist- 
You can add this file jira-config.properties and put the following configuration (jira.index.issue.threads = 20
) while jira 7.1.10 is being indexed, we currently have over 600k records on our instance and it is indexing 2% in 10h. I would like to know if I can do something while the application is indexing to increase its indexing speed.

#file jira-config.properties
jira.index.issue.threads = 20

I will highly appreciate your advice and recommendation.

Thanks in advance for any assistance. :)

[Nic Brough -Adaptavist-]

Decreasing the threads may be a better option, it depends on how your OS handles threads and the CPUs you've got available.

It is a lot more complex than this, but for the fastest indexing, you should try to aim for only 1-3 threads per CPU core to minimise swapping and other overheads.  If I were trying to run a large Jira on this laptop, I'd set the max threads to 11, to give everything else a bit of priority.  On a dedicated server, a few more, as long as the OS has plenty of space to work too.

[Nic Brough -Adaptavist-]

What load is reindexing imposing on your server?

Increasing the threads may not be the right answer, you need to look at why it is slow first.   With 10 threads for example, are your CPUs all busy?  If so, then changing the threads is going to do precisely nothing - there's no CPU to parallelise the work on to.

[Sha Aquino]

Hi Nic,

First issue we encountered was our background reindex takes up to 48 hours to complete which is why we are trying this step based on the page I have mentioned.

While working on this issue, another issue arises and now our background re-index keeps on failing and does not go beyond 2%.

Upon reviewing our logs, we found 2 causes.

1. Server connection time outs

2. We are getting Error getting the next result (The result set is closed)

Upon research on #2, I came across this - https://confluence.atlassian.com/jirakb/jira-re-indexing-fails-with-error-getting-the-next-result-this-resultset-is-closed-346947669.html

Workaround in this article is to update the maxqueue size to 4000 but it can be applied to versions 8+ to mitigate the issue. Unfortunately, we are still on 7.13.0 and are unable to upgrade our version and unable to apply the workaround so we resorted to the last entry which was to increase our pool-max-size.

With the increase to pool-max-size we thought we could try increasing the reindex speed as well to try and mitigate #1.

Thanks in advance for any recommendation you may have for us.

[Nic Brough -Adaptavist-]

You need to monitor what the server is doing to try to find out where the bottleneck is - what are the CPUs, memory and disk doing during the re-index?  Which one(s) of them are overloaded?

[Sha Aquino]

Hi Nic,

This is our numbers while background reindex is running.

As per our infra team, these numbers are okay.

Let me know what you think.

[Nic Brough -Adaptavist-]

That looks healthy, have you tried increasing the memory available to Jira?  (If you have not, then be warned that as a Java system, it's not a good idea to go up in huge chunks.  Don't take an 8Gb Jira up to 16Gb, add 1-2 Gb at most in one step)

[Sha Aquino]

Hi Nic,

No, we have not yet. We will try doing it in increments and see how it goes. I'll update this thread with what happens.

Thanks

[Normann P_ Nielsen _Netic_]

hi @Sha Aquino 

Did Your ever find any good solutuions. We are facing 20 hours for 100% indexing.

I have tried a lot of the "tricks" in the book, but the time/speed keep being up there in the 18-20 hours (thats background indexing)

[Sha Aquino]

Hi @Normann P_ Nielsen _Netic_ 

Unfortunately no. I think it still really depends on the amount of data it needs to reindex. We did increase our memory though 500GB added which really helped make sure our background reindex continues after 5%. Before our increase I kept on getting time out errors in the logs when our reindex fails.

[Nic Brough -Adaptavist-]

The only way to "fix" this is to improve your hardware and reduce the volume of issues that need to be indexed.

[Normann P_ Nielsen _Netic_]

Im not sure the answer is that simple after all :-) 

Anyways, our Jira takes 8-10 hours to index, and I have done som testing yesterday:

https://www.mos-eisley.dk/display/ATLASSIAN/Jira+Reindex+test

In general, nomatter what I did, even safe mode - the difference in time was not much. There are problems with the tests of cause, I never let them finish at all, but the environment as such was the same each time. a clone of our production.

Not done, and I do think the main issue is the number of custom fields, and a clean up and scoping of context has started .... but we do want to use custom fields

I will update this post later on.

BR,

Normann