Data Center Admins: How We Cut Jira Reindex Time from 8 Hours to 35 Minutes

[#1]

If your Jira Data Center instance has over 1 million issues, full reindexes are painful. Ours used to run 8 hours and block users. Here is what actually moved the needle for us.

The problem:
Default settings are tuned for small instances. On Data Center with 2 nodes, 1.4M issues, and 200 custom fields, a foreground reindex locked the system all day.

What we changed:

Increase indexing threads
File: jira dot properties on each node
Change: jira dot indexing dot thread dot count equals 8
Default is 1. We set it to half our CPU cores per node. Always test in staging. Too high and you get contention.
Move to SSD and split the index from home
File: setenv dot sh on each node
Change: JVM_SUPPORT_RECOMMENDED_ARGS equals -Dcom dot atlassian dot jira dot localhome dot indexdir equals /opt/jira-index
We put the index on a dedicated NVMe drive. Latency dropped 60 percent. Never keep index on NFS.
Clean custom fields before reindex
We had 40 unused custom fields with global contexts. We used the app ScriptRunner to find them, then deleted or narrowed the context. Each global custom field adds time to indexing. After cleanup, index size dropped 22 percent.
Use background reindex for apps only
Full foreground reindex is rare now. For app installs, we run background reindex with Issue Indexing Replication. Users stay online.
Result:
Full foreground reindex: 8 hours 10 minutes down to 35 minutes
Background reindex: 2 hours down to 18 minutes
Zero user complaints during business hours

Biggest gotcha:
If you cluster, set the same indexing thread count on all nodes. Mismatched values cause replication lag.

What did you do to speed up indexing. Share your jira dot properties tweaks below.

[Randy McElroy]

This is good info to look at.  What we did to get around the problem was add a node dedicated to indexing.  We have this node excluded from the rotation in HAProxy, but we can then log into that node directly to run the index.  Users never even notice when we are indexing.  Since there are no users on it the time is not an issue.  I haven't timed it but it is fairly quick and much shorter than when we used a shared node.

[Gustavo Chaves]

I recently upgraded Jira from 10.3 to 11.3 and took the opportunity to offload its indexing to an OpenSearch service. After that, the total time for a background reindex got from 8 hours to less than 2 hours.

I have OpenSearch running in the same server as Jira, but we had to increase the server's CPU and memory resources in order to accommodate it.

[Viswanathan Ramachandran]

Those are impressive. Usually, the below delivers big returns

• Remove unused custom fields.
• Replace global contexts with project-specific contexts.
• Archive inactive projects.
• Remove obsolete issue types and workflows.
• Review marketplace apps that add indexers or custom searchers.

The other option could be

• database optimisation (you may have to engage DBAs for expert advice)
• dedicated index storage (use local SSD storage) or NVMe like you mentioned, plus monitor disk IOPS and latency during reindexing.

• tuning/increasing jira.indexing.thread.count can help
  FYI : More threads do not always mean faster indexing. Beyond a certain point, contention can actually increase reindex duration.

[Marco Thamm]

Hey, i want to try this but your post is very hard to read. Where can i find this file? Its not there in my deployment. It seems like it must be called jira-config.properties. Also from your post i dont know what exactly i need to turn here as a setting...

Could you clarify the settings which need to be changed?

[Viswanathan Ramachandran]

hi, you add this as JVM system property in setenv.sh. You will add something like 

-Djira.indexing.thread.count=8

 If jira.indexing.thread.count is not set, Jira automatically chooses an appropriate number of indexing threads based on the server’s available processors and internal defaults.

In most environments, Atlassian recommends leaving it unset unless you’ve identified a specific performance issue and tested the impact of changing it. Like I mentioned more threads doesn't always mean faster reindexing. I would recommend you check with Atlassian support and discuss within your SMEs.

FYI, there a no way to determine the value Jira is using, atleast not through the UI.

Your options are:

• Enable DEBUG logging for Jira’s indexing components (if you’re troubleshooting).
• Inspect startup or indexing log messages, as some Jira versions log details about indexing workers during a re-index.
• Use profiling or thread dumps during a re-index to observe how many indexing worker threads are active.

Hope it helps.