Our indexes were corrupt and we are in process of rebuilding our indexes (we deleted indexes, cache from jira-home) and we see that only 20k issues have reindexed (in 5hours) out of 1015221 issues. Currently our instance is up and users are able to create issues. However, we cannot see issues on the boards, dashboards, or in issue navigator. And, we think that indexing is going to take 15-17hours to complete.
From my previous experience I know that scripted fields make the whole indexing processing slow and I would like to know what steps we can take to speed up the process. All of our 10k users are affected right now and we would like to know and understand what would be the right measures to take to resolve this issue.
We have 50+ scripted fields and the reindexing is going on for 5hours now. Can we disable scriptrunner while background indexing is ongoing?
Thank you for the help.
Technically, you can turn it off while re-indexing.
But that will break every scripted field you've got. The indexing throws away the index for the field and re-creates it. If the scripted field script is not run during index, it won't create the data for the field.
You'll basically destroy all your scripted fields, and the only way to get them back is to run a full re-index with Scriptrunner enabled.
Thank you for prompt reply @Nic Brough _Adaptavist_ . Some of the scripted fields are scoped to "all projects", how would it affect the indexing if I change the scope of these fields?
I also see that sometimes it takes 5hours to reindex 100issues and all of the sudden I see a big jump in few minutes (for example just now it took 5mins to reindex 1000issues), can you please tell me why you think this is happening?
It will, a little.
It will also destroy your ability to search or sort on the field until you re-enable it and do a full re-index.
Disabling bits of it is not the answer here. If you turn off bits of the indexing, you reduce or destroy the usefulness of your scripted fields.
Before I write a long if/then essay, can you tell me what version of Jira you are on? Some versions index in very different ways.
It's been 7hours since we started a background indexing and we are still sitting at 25k issues our of 1015221. I am just not sure what we need to do inorder to get all of our issues back since a background indexing is in process. Any suggestions would be highly appreciated. Thank you.
There's nothing to do but wait at this point.
8.5 has many index improvments in it that save me having to write an essay on a lot of things you can do to improve performance. Last time I upgraded a large 7.13 system which was indexing at 2 hours, we went to 8.13 and the indexing, dropped to a shade under 35 minutes.
So, some thoughts:
I'm worried that something is very very slow on your system - given 300k issues in 35 minutes, I wonder what you're running it on. Is your index on a shared file system? Is there a slow network connection to the database? Low powered CPU or little RAM etc?
We did start with looking at scripted fields, but if it's the scripts for them, then we have to look at what they are doing - why are they taking so much time?
There is one thing to try, but it's a little complex to explain and might make no difference. If you have fields (scripted or not) that are not in use in a project or issue type because you have hidden them in the field configurations or not put them on any screens, you might want to change their "context" as well. Hidden fields are still indexed, including running your scripted field. If you change the field context to remove them from projects, then the indexing will not bother to look at them for the issues in those projects, and hence won't run the scripts.
Thanks again Nic. I will touch base with my team to understand about the cpu/ram etc (as far as I know we are running on aws cloud), and as per our cloud team who manages our server we have enough disk space, and cpu usage < 5%. So I would think there shouldn't be a problem. We also have bigpicture plugin installed on our instance , not sure if it can cause any slowness. You are absolutely right about changing the contexts of our scripted fields..I was planning to do that cause the indexing stats from our logs did show top 10 custom fields where most of them were scripted fields.
I started changing of the scripted fields and I got a pop up asking for a reindex, and to my surprise I see this error in the indexing UI "Jira is unable to perform a background re-index at this time because the index files are either missing or corrupted."(backgtound reindexing option grayed our) I am now confused if the background reindexing is actually running or is it not. WHat do you think?
Space is probably not the issue here there are some basic checks that yell "disk space error" well before you run into a real problem with it. But if your filesystem is a shared one, it's going to be horribly slow (i.e. useless).
CPU and by implication, memory, sounds fine from what you have said.
Indexing stats are a great place to look - whilst I know they can tell us that specific fields are an issue, it would be even better if they can tell us which script it is.
And for the error you now get - yes, you already have a re-index running. The change of custom field setup needs another one to be run, but you're in the middle of one already.
Whilst this is a right pain, it will not do any damage. Wait for the current one to finish, then run another one.
The current one is still running (we are still sitting at 25k issues out of 1mil and we don't see any progress anymore) and I see same errors in the logs "
"2020-12-15 02:15:18,159+0000 SP-bigpicture-0000000.DefaultEventBusSingleJobExecutor-35558 ERROR anonymous [c.a.j.p.s.s.listener.impl.DefaultJiraSlackEventListener] Error processing event
What are my options now? Can you please suggest something Nic.
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events