nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Re-index after 8.5.1 upgrade taking 16days

jeremy.taylor

I'm upgrading 7.13.1 to 8.5.1. The upgrade goes fine but the post upgrade full re-index is taking forever, like on pace to take 16 days. I've tried increasing the index threads, increasing jvm memory, and upping the database connections, nothing is making a difference. I disabled AV (Windows Defender) and confirmed the disk access speeds improved, but that still didn't have an impact on the re-index. The log is showing a few "Script field failed on issue: xxxx" errors but nothing serious. It looks like a lot has changed with indexing in version 8. Anyone know any other tips, tweaks, tuneup options?

Accepted answers

Petr Vaníček

Hi @jeremy.taylor ,

as @Tinker Fadoua mentioned - it's possible that this problem is related to scripts.

So first try to repair scripts (not only imports on first lines of scripts). As I can suggest you didn't check this part of documentation - Preparing for Jira 8.0 - I guess corectly?

In Jira 8 there has been moved ComponentManager and you should start using ComponentAccessor - also Scriptrunner documentation can help you - here.

Try that and let us know, because if there is some kind of problem like that than that can slow down indexing.

All comments

Tinker Fadoua

@jeremy.taylor

That is very odd!!! Actually indexing is one the features that has been improved in Jira 8.x.x

What are you seeing on the logs? How big is your instance? How big is the Data folder?

This is not a normal situation.

Best,

Fadoua

jeremy.taylor

In the log I'm seeing a fair amount of "Script field failed on issue: xxxx" errors, but didn't think that should cause the lag I'm seeing. Instance is pretty large >2 million issues, data folder is around 30GB. Before upgrading a background re-index could take up to 10-12 hours, I expected something on par with that. I let this run for 35 hours and it only got to 10%.

Tinker Fadoua

@jeremy.taylor

I understand for a big instance an indexing will take time, however not running for days 🤔. It will slow Jira as well. Are you running the background one? or the one that locks Jira?

What are those scripts for? You will have to dig little bit inside your instance, remove any unneeded custom fields, workflows,... you need to cleanup.

Then run the full indexing that locks Jira.

I am not advertising here for any vendor however there is a plugin called Optmizer for Jira that can help you a lot.

Did you check the best practices from Atlassian for an instance with this size?

Best,

Fadoua

jeremy.taylor

Yes, running the lock out re-index, apparently the upgrade from 7.x to 8.x invalidates the existing index and requires a full rebuild.
The errors I'm seeing in the log are things like this

2019-11-13 21:17:31,802 Caesium-1-3 ERROR ServiceRunner [c.o.scriptrunner.customfield.GroovyCustomField] *************************************************************************************
2019-11-13 21:17:31,817 Caesium-1-3 ERROR ServiceRunner [c.o.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: AER-2060165, field: Business Complexity
java.lang.NullPointerException: Cannot invoke method multiply() on null object
at Script2.run(Script2.groovy:17)
2019-11-13 21:17:32,145 Caesium-1-3 ERROR ServiceRunner [c.o.scriptrunner.customfield.GroovyCustomField] *************************************************************************************
2019-11-13 21:17:32,145 Caesium-1-3 ERROR ServiceRunner [c.o.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: AER-2060165, field: Service Line (Extended)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script3.groovy: 1: unable to resolve class com.atlassian.jira.ComponentManager
@ line 1, column 1.
import com.atlassian.jira.ComponentManager

I haven't done any Jira administration for a number of years, I'm helping out one of our sister company's with this. If these script errors are causing the issue I'm not sure how to resolve them.

What's bugging me though is I tuned several config options and have not seen any difference in resource utilization. I increased the Jvm memory, upped xmx from 2GB to 8GB and increased the index threads with the following lines in jira-config.properties
jira.index.issue.threads=50
jira.index.sharedentity.threads=50
I expected to see tomcat8.exe using more cpu and memory after making those changes but there was no noticeable affect.

Petr Vaníček

Hi @jeremy.taylor ,

as @Tinker Fadoua mentioned - it's possible that this problem is related to scripts.

So first try to repair scripts (not only imports on first lines of scripts). As I can suggest you didn't check this part of documentation - Preparing for Jira 8.0 - I guess corectly?

In Jira 8 there has been moved ComponentManager and you should start using ComponentAccessor - also Scriptrunner documentation can help you - here.

Try that and let us know, because if there is some kind of problem like that than that can slow down indexing.

Tinker Fadoua

Yep @Petr Vaníček there are lot of troubleshooting steps to find the reason behind Jira extreme slowness.

@jeremy.taylor This is really an unusual behavior, find out what are those scripts for will be there first step.

Let us know if you have any questions. I would love to help you.

Also check this document out regarding sizing recommendations

Best,

Fadoua

jeremy.taylor

Thanks guys, I'll have to refer back to the developers on those scripts. I'm just a DevOps Admin on loan for this, I don't know the details of their environment very well, and I haven't run a Jira instance myself in 10 years. I did look back at the old logs and see those same script errors were occurring before I even did the upgrade. So I don't think they're causing the issue.

Petr, I don't know if they read that document and did any preparation in that regard. I went through this document to perform the upgrade. All the update checks in Jira passed and the install itself went fine.

I can really only speak here from an infrastructure standpoint, what has me confused is why the tuning parameters I changed have had no impact? I would think increasing the index threads from 10 to 50 would have a noticeable impact on CPU utilization, and increasing the JVM memory would show more ram utilization. I'm not seeing any change no matter what I set those values to.

Tinker Fadoua

@jeremy.taylor

Please check the link I sent you.

There must be something over using the system resources. Can you please check Catalina log? Let's see what's in there.

Best,

Fadoua

Jens Kisters __SeibertSolutions

I have stumbled over the same problem even though i already made my scripts be compatbile with Jira 8, but some scripts fail because foreground indexing is in progress.

/secure/admin/IndexReIndex!reindex.jspa [c.o.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: ERG-3715, field: Summary neues Ticket
com.atlassian.jira.issue.index.SearchUnavailableException
at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:885)
at com.atlassian.jira.issue.index.DefaultIndexManager.getIssueSearcher(DefaultIndexManager.java:865)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
at com.sun.proxy.$Proxy8.getIssueSearcher(Unknown Source)
at com.atlassian.jira.issue.search.SearchProviderFactoryImpl.getSearcher(SearchProviderFactoryImpl.java:17)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getIssueSearcher(LuceneSearchProvider.java:130)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:443)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:450)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:439)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:218)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:362)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:135)
at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:140)
at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118)
at com.atlassian.jira.bc.issue.search.SearchService$search$0.call(Unknown Source)
at ScriptedFieldTicketSummary.run(ScriptedFieldTicketSummary.groovy:57)

I will try the following procedure:

uninstall script runner before the upgrade
do the upgrade
install script runner again
do a background index

Tinker Fadoua

@Jens Kisters __SeibertSolutions let us know how it goes.

Best,

Fadoua

Jens Kisters __SeibertSolutions

@Tinker Fadoua :
i tried this today:

uninstall script runner before the upgrade
do the upgrade
install script runner again
do a background index

and it worked well.

With Script runner active:
Foreground reindex had an estimated duration of 5 hours, but reindex aborted at 61%.

Disabled Script runner, ran full foreground index in 25 minutes, activated script runner and the background index took another 35 minutes.

I think the problem is:
A scripted field tries to run a search.
SearchService is unavailable during full reindex and this produces a exception, probably after a timeout.

Tinker Fadoua

@Jens Kisters __SeibertSolutions How is the instance running? Everything good?

Best,

Fadoua

Jens Kisters __SeibertSolutions

Hey @Tinker Fadoua

the solution worked fine.

Cheers

Jens

Jens Kisters __SeibertSolutions

By the way i just stumbled over some documentation how to handle this error:

https://scriptrunner.adaptavist.com/latest/jira/scripted-fields.html?_ga=2.205056404.502615708.1589186444-515719914.1542286934#_jql_searches_in_script_fields

Rishikesh Dugyala

@jeremy.taylor you can disable scripted fields module under script runner plugin or you can disable scriptrunner plugin before reindex after completion of reindex you can turn back on scripted field or script runner plugin this reduces more time while reindexing.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

server

jira

jira-server