How can I reduce full re-index time after upgrade Jira v9? (It tooks almost 57hours.)

[Seonghwa Hong]

Hi there,

The Jira Data Center operated by our company is 8.22.6 version and is clustered into two nodes. Account is used by Crowd. (Jira Licensed users are approximately 1,900 users, All Crowd AD are approximately 80,000 users) And There are about 396,000 issues. 15 plug-ins have been installed, and 3 self-developed plug-ins have been installed and used.
And our server specification is as follows.

DB server:

• 16Cores
  RAM 48GB
  Disk 1,000GB

Jira server:

• 16Cores
  RAM 48GB
  Disk 2,500GB

We were upgraded the Jira in the same copied development environment as the production environment. After upgrade ended, full re-index is worked automatically and I found the log as below when the upgrade is complete.

[c.a.j.issue.index.DefaultIndexManager] ReindexAll took: 206545129 ms in the foreground, index size is 2 GB

Full re-index took almost 57 hours.

Almost all custom field in our Jira use the global field context, so seems to be expensive indexing time as Atlassian documents said. Even so, I wonder if it is normal to take 57 hours. It takes too long.

We going to upgrade Jira at Saturday, but If the reindex takes 57 hours then reindexing is completed early Tuesday dawn, our employee will not be able to use Jira at Monday.

I would like to full reindex without downtime (or less than 30 hours of reindex time) after the version upgrade. Can I copy and use the index directory of the previous backup data? (We have a backup file dated 11th Oct that upgraded version 9 as I said above, and created indexesV2/snapshots directory.)

Is there a good way to reduce the time of re-index? Or let me know if you have any good experiences. Thanks in advance for any help.

[J C]

Were you able to find a solution for this?  I am running into the same problem. I've done some adjustments to the underlying instances, toggled some dbconfig and indexing settings and I'm at 20 minutes for 1% progress average.  I don't want to throw more hardware at it but the CPU on the database server is spiked at 100% and the CPU on the node fluctuates but nowhere near 100%.

[J C]

The best I was able to do was get it down to 4-5 min for 1% of progress resulting in about 8 hours of indexing time by updating the hardware significantly.  That would not be a problem if it was just during the ugprade but it looks like subsequent re-indexes perform the same way.

[David Bakkers]

Hello @Seonghwa Hong 

What's wrong with following Atlassian's well documented advice and indexing one node while it's not part of the load balancer group, then adding it back to the group and allow the two nodes to syncronise themselves?

If you don't want to do that, what's wrong with following Atlassian's other well documented advice and allocate more threads / CPUs while doing the re-index?

If you're not prepared to follow either of those VERY WELL DOCUMENTED METHODS, then the time it takes is the time it takes, and, as you probably already know yourself from just reading the documentation...

This can take anywhere from seconds to hours, depending on the number of issues and comments in your Jira instance. While re-indexing is taking place, your instance will be unavailable to all users unless you choose Background Indexing

[Seonghwa Hong]

Hi David,

Because after I shut down both nodes, I must upgrade only one node and proceed with the full re-index. (Version 8 and Version 9 have different index files, so it is impossible to the background indexing after upgrading to 9.)

Then, the test results show that the full re-index takes 57 hours, so if i  perform the upgrade on Saturday, the full re-index after upgrade will be completed on Monday evening, there is a problem with employees not being able to use Jira on monday.

So I'd like to know how to reduce the full re-index time to less than 30 hours. Or I'd like to know if I can use the index file from the previous date (the file indexed as a result of the test upgrade).

[J C]

The upgrade from 8 to 9 causes a significant incease in upgrade time due to the changes in the indexing architecture.  Although Atlassian does have very well documented methods, the upgrade to 9.4.x and 9.12.x is not as straight forward as previous upgrades because in 9.12.x the system will not be available and remain in maintenance mode until a valid index is available, you can have flags and etc set to let you into the system before you run the index but once you start the index, the system is unsuable and if it takes over 24 hours... that's a problem.  My previous index in 8.x only took 1 hour, there's something fundamentally wrong if in 9.x it takes up to 24x more time than my previous index with the same settings.

[Noni Khutane]

@J C Were you able to come up with a solution?

[J C]

@Noni Khutane  - Ah yes, we are running Postgres, and hence 8 to 9 requires an update to the database, after running analyze and reindex on the database right after the database upgrade, the indexing was at normal speed, finished in about an hour.