Can we disable scriptrunner while background indexing is ongoing?

[sachin gangam]

Our indexes were corrupt and we are in process of rebuilding our indexes (we deleted indexes, cache from jira-home) and we see that only 20k issues have reindexed (in 5hours) out of 1015221 issues. Currently our instance is up and users are able to create issues. However, we cannot see issues on the boards, dashboards, or in issue navigator. And, we think that indexing is going to take 15-17hours to complete.

From my previous experience I know that scripted fields make the whole indexing processing slow and I would like to know what steps we can take to speed up the process. All of our 10k users are affected right now and we would like to know and understand what would be the right measures to take to resolve this issue.

We have 50+ scripted fields and the reindexing is going on for 5hours now. Can we disable scriptrunner while background indexing is ongoing?

Thank you for the help. 

[Nic Brough -Adaptavist- · Accepted solution]

Technically, you can turn it off while re-indexing.

But that will break every scripted field you've got.  The indexing throws away the index for the field and re-creates it.  If the scripted field script is not run during index, it won't create the data for the field. 

You'll basically destroy all your scripted fields, and the only way to get them back is to run a full re-index with Scriptrunner enabled.

[sachin gangam]

Thank you for prompt reply @Nic Brough -Adaptavist- . Some of the scripted fields are scoped to "all projects", how would it affect the indexing if I change the scope of these fields?

I also see that sometimes it takes 5hours to reindex 100issues and all of the sudden I see a big jump in few minutes (for example just now it took 5mins to reindex 1000issues), can you please tell me why you think this is happening?

[sachin gangam]

Does changing "Searcher: Free Text Searcher " to "none" help fasten the current background indexing process?

[Nic Brough -Adaptavist-]

It will, a little.

It will also destroy your ability to search or sort on the field until you re-enable it and do a full re-index.

Disabling bits of it is not the answer here.  If you turn off bits of the indexing, you reduce or destroy the usefulness of your scripted fields.

Before I write a long if/then essay, can you tell me what version of Jira you are on?  Some versions index in very different ways.

[sachin gangam]

Thank you Nic, we are on Jira Server 8.5.5

[sachin gangam]

It's been 7hours since we started a background indexing and we are still sitting at 25k issues our of 1015221. I am just not sure what we need to do inorder to get all of our issues back since a background indexing is in process. Any suggestions would be highly appreciated. Thank you. 

[Nic Brough -Adaptavist-]

There's nothing to do but wait at this point.

8.5 has many index improvments in it that save me having to write an essay on a lot of things you can do to improve performance. Last time I upgraded a large 7.13 system which was indexing at 2 hours, we went to 8.13 and the indexing, dropped to a shade under 35 minutes.

So, some thoughts:

I'm worried that something is very very slow on your system - given 300k issues in 35 minutes, I wonder what you're running it on.  Is your index on a shared file system?  Is there a slow network connection to the database?  Low powered CPU or little RAM etc?

We did start with looking at scripted fields, but if it's the scripts for them, then we have to look at what they are doing - why are they taking so much time?

There is one thing to try, but it's a little complex to explain and might make no difference.  If you have fields (scripted or not) that are not in use in a project or issue type because you have hidden them in the field configurations or not put them on any screens, you might want to change their "context" as well.  Hidden fields are still indexed, including running your scripted field.  If you change the field context to remove them from projects, then the indexing will not bother to look at them for the issues in those projects, and hence won't run the scripts.

[sachin gangam]

Thanks again Nic. I will touch base with my team to understand about the cpu/ram etc (as far as I know we are running on aws cloud), and as per our cloud team who manages our server we have enough disk space, and cpu usage < 5%. So I would think there shouldn't be a problem. We also have bigpicture plugin installed on our instance , not sure if it can cause any slowness. You are absolutely right about changing the contexts of our scripted fields..I was planning to do that cause the indexing stats from our logs did show top 10 custom fields where most of them were scripted fields. 

[sachin gangam]

I started changing of the scripted fields and I got a pop up asking for a reindex, and to my surprise I see this error in the indexing UI "Jira is unable to perform a background re-index at this time because the index files are either missing or corrupted."(backgtound reindexing option grayed our) I am now confused if the background reindexing is actually running or is it not. WHat do you think?

[Nic Brough -Adaptavist-]

Space is probably not the issue here there are some basic checks that yell "disk space error" well before you run into a real problem with it.  But if your filesystem is a shared one, it's going to be horribly slow (i.e. useless). 

CPU and by implication, memory, sounds fine from what you have said.

Indexing stats are a great place to look - whilst I know they can tell us that specific fields are an issue, it would be even better if they can tell us which script it is.

And for the error you now get - yes, you already have a re-index running.  The change of custom field setup needs another one to be run, but you're in the middle of one already.  

Whilst this is a right pain, it will not do any damage.  Wait for the current one to finish, then run another one.

[sachin gangam]

The current one is still running (we are still sitting at 25k issues out of 1mil and we don't see any progress anymore) and I see same errors in the logs "

"2020-12-15 02:15:18,159+0000 SP-bigpicture-0000000.DefaultEventBusSingleJobExecutor-35558 ERROR anonymous     [c.a.j.p.s.s.listener.impl.DefaultJiraSlackEventListener] Error processing event

java.lang.NullPointerException"


What are my options now? Can you please suggest something Nic. 

[sachin gangam]

hey Nic, after I disabled scriptrunner and bigpicture it took 50mins for a full lock reindex. Now we are backup and running. 

[Nic Brough -Adaptavist-]

Yep, that makes sense, but it does of course mean all your scripted fields and big-picture stuff is now useless.

[sachin gangam]

You mean they are damaged? and doesn't display on the issues? 

[Nic Brough -Adaptavist-]

Possibly incorrect or simply missing data completely, yes.

You'll need to do a full re-index with the apps enabled to fix them, or edit/update every issue they are used with.

[Ryan Trondsen]

I have noticed that during a full re-index, any scripted field that does a JQL query in the script will cause errors in the log, and will slow the re-index to never finish. We have to disable those fields before running the full re-index, and then re-enable those fields, and run a localized re-index on the issues that use those scripted fields. In general, scripted fields and BigPicture don't cause massive delays. We generally only run background re-indexes, and haven't had issues. Perhaps try disabling individual fields and testing your reindexing to see if a particular one is causing issues. I seriously doubt BigPicture is doing it.

[Nic Brough -Adaptavist-]

That smacks of very poor scripts, ones trying to do too much, or, more likey, recursing - what are they doing in the searches?  Would the read of the results be calling another search which calls another search which calls another search....

[Ryan Trondsen]

When using the SearchService in a Scripted Field, it fails to re-index during a "Full" reindex only (ie when the index is deleted) as the script tries to search the index). An example function I use is this:

// Find issues method, use: findIssues("JQL Search here")
def findIssues(String jqlSearch) {
    def user = ComponentAccessor.jiraAuthenticationContext.loggedInUser
    def searchService = ComponentAccessor.getComponentOfType(SearchService)
    def parseResult = searchService.parseQuery(user, jqlSearch)
    if (!parseResult.isValid()) {
        return null
    }
    def results = searchService.search(user, parseResult.query, PagerFilter.unlimitedFilter)
    return results.results
}

 I generally avoid using JQL search scripts in scripted fields whenever possible though.

Again, this is only during full re-indexes. We have 200k+ issues and have 2-3 hour max background re-indexes.

[Nic Brough -Adaptavist-]

Hmm.  That would make some sense - the index call will cause a search which will fail because the index it's reading might not be there.  Slightly recursive.   However

1) It should just fail and leave the scripted field with a duff index, not massively slow down

2) On a small-medium system, I just tried it and got a handful of "can't read index" and a lot of "no results from search, so returning nothing" back.

I can't say anything about speed, it took 15 minutes before I added the script, 12 afterwards, but the number of issues the field is valid for isn't huge.