Re-index stuck at 95% - what to do?

Question

Long story but we managed to create a 'ghost' index for an issue which screwed up the searches by causing a NullPointerException.

To fix the indexes we re-indexed, which we often do.  The re-index usually takes around 13 minutes.  However this time it chugged along happily up to 95% as usual and then sat there, not as usual, no error messages in the logs, only the 'i18n problem: id defined by localised value: Refresh' message each time my browser page refreshed.

After 10 minutes of sitting at 95% I restarted the JIRA service (ouch! but didn't know what else to do and our users needed to get back in).

JIRA started OK, the bad index had been fixed and searches for filters and dashboards also seem to be working, but the 're-index recommended' message still appears on the admin pages for a custom field change made 2 days ago.

I've run the Integrity Checker which found no errors.  I've checked Java Melody (we've installed the plugin) which indicates we were OK for memory and DB connections.

So what should I do now?  I don't like leaving JIRA incompletely re-indexed but I'm wary of re-indexing again in case it gets stuck again, and if the next re-index is likely to go on indefinitely I'll have to get the down-time authorised first.  I don't even know if the re-indexing would eventually finish if I wait long enough (how long is long enough???).  Anyone else seen this?  Is there anything special about 95% that might hint at what's going on?  We're running 4.0.2.

J Thomas4 · Accepted Answer

In case it helps anyone else here's what we did:

1) After the bad re-index, searched for the most recent issues using Issue Navigator and compared that to the issues in the database - there were 4 existing issues not found in the Issue Navigator search (ie didn't get indexed).  Edited each of these 4 issues (eg added a comment) to get them re-indexed.  That fixed the immediate problem.

2) Restored a production DB backup on a test instance and tried re-indexing - completed with no problems.  If there had been problems then we could have turned on debugging on the test instance (not an option on prod as we didn't want to risk another flaky re-index to investigate the problem) but this wasn't necessary.

3) Scheduled a time for a production re-index allowing extra time in case of problems, and re-indexed.  It completed without a problem.