Perform Index Recovery on JIRA Startup if current Index invalid.

[Franz Limbacher]

I am working on a fallback if the index of our jira instance after a move/restart is invalid.
The idea was to use the index recovery as it is much faster than the complete rebuild. But I would like to have it automated, so I don't have to look for the snapshot and trigger the recovery.

Via the rest call /rest/api/2/index/summary i get information about the status of the index. But I am not able to trigger the index recovery via rest.
So to execute it from a shell via REST I lack features.

Then I thought about a plugin to perform the index recovery. It should be possible using this Manager:
https://docs.atlassian.com/software/jira/docs/api/7.7.1/index.html?com/atlassian/jira/index/ha/IndexRecoveryManager.html

But I don't know yet, where to find the information/summary about the current index, to find out if it is necessary to perform the recovery.

Does anyone have an idea how to analyse or perform this? Are there any experiences?

Thank you for your help!

[freelimbo · Accepted solution]

Unfortunately I was not able to perform it via rest or a plugin. I now resolved it with a script, that is executed before my server is started. 

I just analyse if the index is older than the latest snapshot. If that is the case I unzip the snapshot into the index folder. This works fine for me:

if [ ! -e $destination/indexes ]; then
  echo "Current index doesn't exist. Auto recovery will be started.";
  ... perform unzip ...
elif [ $zipfile -nt $destination/indexes/$(ls -t $destination/indexes/ | head -n1) ]; then
  echo "Current index is older than latest backup. Auto recovery will be started.";
  ... perform unzip ...
else
  echo "Backup is older than current index. No recovery necessary."
fi

[Taylor Barrett-Wilsdon]

Very curious about the same - would love to hear if you had an update @Franz Limbacher

https://jira.atlassian.com/browse/JRASERVER-37896 also may be of interest to you.

[freelimbo]

Hello @Taylor Barrett-Wilsdon,

Unfortunately I was not able to perform it via rest or a plugin. I now resolved it with a script, that is executed before my server is started. 

I just analyse if the index is older than the latest snapshot. If that is the case I unzip the snapshot into the index folder. This works fine for me:

if [ ! -e $destination/indexes ]; then
  echo "Current index doesn't exist. Auto recovery will be started.";
  ... perform unzip ...
elif [ $zipfile -nt $destination/indexes/$(ls -t $destination/indexes/ | head -n1) ]; then
  echo "Current index is older than latest backup. Auto recovery will be started.";
  ... perform unzip ...
else
  echo "Backup is older than current index. No recovery necessary."
fi

[Mihai Schwarz]

Hi @Franz Limbacher based from my experience, usually reconstructing the indexes can cause more issues than you previously had. I would strongly recommend that you should lock and re-index.

There is an article on how to rebuild the indexes using directly from your server without using API calls: https://confluence.atlassian.com/jirakb/lucene-index-corruption-931237172.html

[Franz Limbacher]

Hi @Mihai Schwarz

Not quite what I was looking for. My issue is the downtime. If i lock and re-index my system is not accessable for the next 2h+. Then I have a state of about 100% correct index.
If I use the index snapshot (zip) generated by jira it only takes a couple of minutes. Then I have a state of about 99% correct index.  Which is fine for me.

Also it is important for me that this is automated. So no manual work for me like checking if the index is broken and then doing the reindex.

[Mihai Schwarz]

Hi @Franz Limbacher at this point I am not aware of any scripts that do that, the only thing you can check and see if the indexes are broken is through Troubleshoot and Support tools and check Instance health. Usually if you follow the best practices you should have broken indexes. I've noticed that sometime you can notice errors regarding broken indexes in the Instance health but those errors disappear after a couple of minutes. Also make sure that you have enough free space on the disk you have your indexes hosted.