What am I doing wrong? Scriptrunner scripted field prevents Full Re-Index

[Janek Schumann]

Dear Community,

Backstory
I am at loss here and seeking help. For 2 month now our Jira Data Center has an indexing problem. We tried twice to run a full re-index, which usually took 30-40 minutes to complete. Now, it doesn't even complete after hours and hours. We had multiple downtimes within our business hours with 1.000+ users.

Root Cause
Today, we think we found the culprit: Scripted fields with Scriptrunner.

Since Scriptrunner is well established, it must be something we are doing wrong / using it wrong.

Pseudo Code
Here is some pseudo code of the field causing issues:

def ISSUETYPE1 = ... 
def ISSUETYPE2 = ... 
def FIELDID1 = ... 
def projectKey = ... 

// search for issues 
try { 
  issues = Issues.search( 
    "issuetype IN (${ISSUETYPE1}, ${ISSUETYPE2}) " +
    "AND cf[${FIELDID1}] = ${projectKey}"
  )
  .stream()
  .toList();
} 
catch (Exception ex) 
{ 
return null; 
} 

// GUARD: none found 
if (issues.size() == 0) { /*do something...*/ return null } 
// GUARD: more than one found 
if (issues.size() > 1) { /*do something...*/ return null } 
// DEFAULT 
return issues.first();

One example of many. This is one of the least complex fields causing the issue.

The problem
While performing a full re-index, the index is first deleted and then rebuild.
While rebuilding the index, the scripted field is queried to add to the index.
This field accesses a search, which in turn accesses the index, which in this case DOES NOT EXIST!

The indexer is waiting for the field to complete.
The field seems to raise an exception, log it, create insane disk I/Os, ...
The indexer throws an exception that is waited for 5000 ms and is still waiting, creating even more disk I/Os for logging

Workaround
We ended up deleting those fields.
before: index recovery was at 30% after 5 minutes and hanging to index the delta changes.
after: index recovery completed in 25 seconds.
We will try a full re-index in 2 or 3 weeks, since we are super afraid of the next yet another downtime. Each downtime costs us around 125k USD. And we had 2 of those already.

Question
Any ideas on how to resolve this? Or should we avoid HAPI search in scripted fields altogether?

are we using the stream/toList wrong? I wonder...


We spent huge time to debug our Jira since the logging is non-existing, not only from Scriptrunner but from almost every plugin. We often see "Scriptrunner throw an Exception" with no hint on where, which line, ... It's the search for a grain of sand in the known universe.

[Nic Brough -Adaptavist-]

Hi Janek,

The problem here is the search part of the script.  Issue search uses the index, so when you don't have one, it's going to thrash the disk looking for it because of the way the indexing sub-system is written.

The only way I can think of to fix this is to include a check to see if there is an indexing in progress.  You can run through what TaskManager.getLiveTasks() returns to see if there is a full, background, or project index in progress, but I cannot remember if you can tell the difference between them.

Obviously, if there is, you can simply exit from the script before it tries to do anything.

You could also just stick an "exit" at the top of the script every time you want to run a re-index (shouldn't be a huge problem - you shouldn't be re-indexing often enough for that to become onerous)

The problem with not indexing the scripted field is that your field will become useless in search and reporting after you do a a mass indexing run.

There are two solutions:

• Look at why you think you need this scripted field - why do your users think they need to see a list of issues of a certain type from another project?  Are they always going to be relevant? 
• Could you convert to using a standard field - an issue link or (read only) text field that could be updated with a listener when someone creates or deletes an issue that matches the search?

[Victor Law]

Hi @Janek Schumann

During re-indexing, the Scripted field executes a search for every Jira issue. However, this is expected to fail during locked-down re-indexing, as the indexes are not operational then. (It is not wrong on the code level, but the search will not work during the locked-down re-indexing)

If possible, avoid scripting a field that performs a search. 

I strongly suggest deeply understanding the field's use case and determining whether its value can be updated and stored during an update instead of being dynamically generated for every Jira issue. (Imagine showing the field in the issue navigator, which returns 1,000 issues in one go, and it will likely cause performance degradation)

Thank you.