nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

What am I doing wrong? Scriptrunner scripted field prevents Full Re-Index

Janek Schumann_qfvkhc

Dear Community,

Backstory
I am at loss here and seeking help. For 2 month now our Jira Data Center has an indexing problem. We tried twice to run a full re-index, which usually took 30-40 minutes to complete. Now, it doesn't even complete after hours and hours. We had multiple downtimes within our business hours with 1.000+ users.

Root Cause
Today, we think we found the culprit: Scripted fields with Scriptrunner.

Since Scriptrunner is well established, it must be something we are doing wrong / using it wrong.

Pseudo Code
Here is some pseudo code of the field causing issues:

def ISSUETYPE1 = ... 
def ISSUETYPE2 = ... 
def FIELDID1 = ... 
def projectKey = ... 

// search for issues 
try { 
  issues = Issues.search( 
    "issuetype IN (${ISSUETYPE1}, ${ISSUETYPE2}) " +
    "AND cf[${FIELDID1}] = ${projectKey}"
  )
  .stream()
  .toList();
} 
catch (Exception ex) 
{ 
return null; 
} 

// GUARD: none found 
if (issues.size() == 0) { /*do something...*/ return null } 
// GUARD: more than one found 
if (issues.size() > 1) { /*do something...*/ return null } 
// DEFAULT 
return issues.first();

One example of many. This is one of the least complex fields causing the issue.

The problem
While performing a full re-index, the index is first deleted and then rebuild.
While rebuilding the index, the scripted field is queried to add to the index.
This field accesses a search, which in turn accesses the index, which in this case DOES NOT EXIST!

The indexer is waiting for the field to complete.
The field seems to raise an exception, log it, create insane disk I/Os, ...
The indexer throws an exception that is waited for 5000 ms and is still waiting, creating even more disk I/Os for logging

Workaround
We ended up deleting those fields.
before: index recovery was at 30% after 5 minutes and hanging to index the delta changes.
after: index recovery completed in 25 seconds.
We will try a full re-index in 2 or 3 weeks, since we are super afraid of the next yet another downtime. Each downtime costs us around 125k USD. And we had 2 of those already.

Question
Any ideas on how to resolve this? Or should we avoid HAPI search in scripted fields altogether?

are we using the stream/toList wrong? I wonder...

We spent huge time to debug our Jira since the logging is non-existing, not only from Scriptrunner but from almost every plugin. We often see "Scriptrunner throw an Exception" with no hint on where, which line, ... It's the search for a grain of sand in the known universe.

Accepted answers

Victor Law - Ricksoft

Hi @Janek Schumann

During re-indexing, the Scripted field executes a search for every Jira issue. However, this is expected to fail during locked-down re-indexing, as the indexes are not operational then. (It is not wrong on the code level, but the search will not work during the locked-down re-indexing)

If possible, avoid scripting a field that performs a search.

I strongly suggest deeply understanding the field's use case and determining whether its value can be updated and stored during an update instead of being dynamically generated for every Jira issue. (Imagine showing the field in the issue navigator, which returns 1,000 issues in one go, and it will likely cause performance degradation)

Thank you.

All comments

Victor Law - Ricksoft

Hi @Janek Schumann

If possible, avoid scripting a field that performs a search.

Thank you.

Nic Brough -Adaptavist-

Hi Janek,

The problem here is the search part of the script. Issue search uses the index, so when you don't have one, it's going to thrash the disk looking for it because of the way the indexing sub-system is written.

The only way I can think of to fix this is to include a check to see if there is an indexing in progress. You can run through what TaskManager.getLiveTasks() returns to see if there is a full, background, or project index in progress, but I cannot remember if you can tell the difference between them.

Obviously, if there is, you can simply exit from the script before it tries to do anything.

You could also just stick an "exit" at the top of the script every time you want to run a re-index (shouldn't be a huge problem - you shouldn't be re-indexing often enough for that to become onerous)

The problem with not indexing the scripted field is that your field will become useless in search and reporting after you do a a mass indexing run.

There are two solutions:

Look at why you think you need this scripted field - why do your users think they need to see a list of issues of a certain type from another project? Are they always going to be relevant?
Could you convert to using a standard field - an issue link or (read only) text field that could be updated with a listener when someone creates or deletes an issue that matches the search?

Janek Schumann_qfvkhc

That's an idea I came up with today as well. There is also the /status REST API, which will return MAINTENANCE if in maintenance, i.e. when a full re-index is running.

And then perform a background re-index after the full re-index.

The overall goal was to get rid of non-supported plugins, copying data with automation rules between issues, ... and replace it by a dynamic query.

Janek Schumann_qfvkhc

I am accepting this answer over Nic's because of the simplicity and being on point:

If possible, avoid scripting a field that performs a search.

That's the core solution.

Not sure if Scriptrunner's documentation contains a hint for this. Haven't found any.
The Syntax check doesn't highlight this for fields to be a potential issue either.

Janek Schumann_qfvkhc

This answer provides a workaround to the problem by modifying the code and checking for this situation.

Viable, but tricky as well.

Janek Schumann_qfvkhc

@Nic Brough -Adaptavist-

I think the main issue with standard fields / how Jira is designed:
what we are doing with those scripted fields is "get me the content of a field of a linked issue".
Jira is only capable of "get me the linked issue" by design.

This might seem like a small step, but increases the complexity to actually get it done quite dramatically.
-> "misusing" Jira for something it is not designed for. That's the root-root cause. 😅

Janek Schumann_qfvkhc

@Nic Brough -Adaptavist- & @Victor Law - Ricksoft

Thanks for your help. Much appreciated! 🙏

Janek Schumann_qfvkhc

Follow-up question:
The built-in field type "Show parent issue in hierarchy", does it suffer from the same issue?

I see two different ways to get where we want to go with a scripted field:
1. JQL with "issuefunction" and "linkedIssuesRecursive"
-> what we did
-> accesses the index
-> FAIL

2. manually iterating the issue object, looking for links, following them, ...
-> accessing Issue data does not access the index, right?
-> WIN?

How is the "Show parent issue in hierarchy" implemented? option 1 or option 2?
(as an example; this question could be extended to any built in field types from ScriptRunner)

Matt Doar _Adaptavist_

https://docs.adaptavist.com/sr4js/9.1.1/features/script-fields/script-field-tips#id-.ScriptFieldTipsv8.30.0-jql says

"We do not recommend running JQL searches in scripted fields. The JQL search index is not available when running a full re-index, causing delays for re-indexes, including JQL search scripted fields."

Janek Schumann_qfvkhc

Thanks for the link, Matt.

😇

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

scriptrunner

jira

jira-data-center

data-center