Jira 8.5.0 full foreground re-index fails

[Sebastian Brudziński]

Hi everyone,

I'm trying to run the full re-index with Jira 8.5.0 but I'm always getting the following error:

The logs are only showing some very generic two errors:

com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:83)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:27)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:18)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:537)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:495)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
at java.lang.Thread.run(Thread.java:748)

and

2019-11-06 17:18:56,078 JiraTaskExecutionThread-12 ERROR admin 1034x38539x1 w2zzv0 213.192.95.26,10.0.128.75 /secure/admin/jira/IndexReIndex!reindex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] 
Reindex All FAILED. Indexer: DefaultIndexManager: paths: [/var/atlassian/application-data/jira/caches/indexesV1/comments, /var/atlassian/application-data/jira/caches/indexesV1/issues, /var/atlassian/appl
ication-data/jira/caches/indexesV1/changes, /var/atlassian/application-data/jira/caches/indexesV1/worklogs]
org.ofbiz.core.util.GeneralRuntimeException: Error getting the next result (This ResultSet is closed.)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:272)
at com.atlassian.jira.ofbiz.DefaultOfBizListIterator.next(DefaultOfBizListIterator.java:100)
at com.atlassian.jira.ofbiz.WrappingOfBizListIterator.next(WrappingOfBizListIterator.java:68)
at com.atlassian.jira.issue.util.DatabaseIssuesIterator.pullNextIssue(DatabaseIssuesIterator.java:97)
at com.atlassian.jira.issue.util.DatabaseIssuesIterator.populateNextIssueIfNull(DatabaseIssuesIterator.java:68)
at com.atlassian.jira.issue.util.DatabaseIssuesIterator.hasNext(DatabaseIssuesIterator.java:51)
at com.atlassian.jira.issue.util.DatabaseIssuesIterable.foreach(DatabaseIssuesIterable.java:55)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:335)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.indexIssuesBatchMode(DefaultIssueIndexer.java:182)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.indexIssuesBatchMode(DefaultIssueIndexer.java:198)
at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1023)
at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:1002)
at com.atlassian.jira.issue.index.DefaultIndexManager.lambda$reIndexAll$0(DefaultIndexManager.java:331)
at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:378)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:331)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:264)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
at com.sun.proxy.$Proxy12.reIndexAll(Unknown Source)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:66)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:27)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:18)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:537)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:495)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: This ResultSet is closed.
at org.postgresql.jdbc.PgResultSet.checkClosed(PgResultSet.java:2696)
at org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:1769)
at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:191)
at org.ofbiz.core.entity.EntityListIterator.next(EntityListIterator.java:266)

I've tried re-running it several times and clearing JIRA_HOME/caches but nothing helped. It also always crashes at step: Currently working on: PortalPage

Any idea what might be the cause, what might be worth checking, or at least how to go forward to figure out why the re-index is failing?

Best regards,
Sebastian

[Pablo Barroso Cubillas]

I was suffering from same error.

Tried every solution / workaround proposed around the web for the same or similar error.  Finally, as I suspected there was some trouble happening with the database conn pool, kind of a disconnection, I changed a parameter within Tomcat Jira config ( <pool-remove-abandoned-timeout> ) and voila: indexing working again.

Within dbconfig.xml, increase the timeout for abandoned connections:

From this:

<pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout>

to this:

<pool-remove-abandoned-timeout>3600</pool-remove-abandoned-timeout>

Please take in consideration your index size needs:  My  Full re-index operation takes 40 minutes, so, 3600 seconds is ok for me, but ymmv.

[Vasilii Kizhaev _Appfire_]

Worked for me! Thanks, Pablo!

[Vasilii Kizhaev _Appfire_]

Unfortunately, it takes 20h to re-index instead of ~1h in the user guide. Has anyone had the same problem?

[Prashant Mali]

any resolution on this. i am also facing the same issue.

[Mike Rathwell]

@Sebastian Brudziński , I am going to take a total stab in the dark based on seeing the PGSql errors in this thread.... Maybe run through the various bits of the Integrity Checker one by one to see if there are any oddities in there before trying to re-index again.

It feels like the DB is inconsistent and it's not necessarily borking on the index as not able to get the right stuff out of the DB.

The only weirdness I got when I first trialed going to 8.5.0 was some Groovy stuff still pointing at the moved ComponentManager but once corrected, the migration in test and then prod was flawless.