Indexing fails with Reindex All FAILED error

Every week we performs a JIRA Background Re-Indexing and most of the time it fails at 97% with the below error. 

 2018-06-09 16:57:53,355 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 93% complete. Current index: Issue
2018-06-09 17:02:38,574 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 94% complete. Current index: Issue
2018-06-09 17:07:17,169 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 95% complete. Current index: Issue
2018-06-09 17:11:54,462 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 96% complete. Current index: Issue
2018-06-09 17:16:33,357 ISSUE-indexQueue:thread-1 INFO gec-jirasvc [action.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: Issue
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:17:41,441 http-bio-8081-exec-12904 WARN anonymous 1037x11155008x2 - /plugins/servlet/streams [auth.trustedapps.filter.TrustedApplicationFilterAuthenticator] User 'vn0l52t' referenced by trusted application: 'jira:8098344' is not found.
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:42,044 JiraTaskExectionThread-508 INFO abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [jira.issue.index.DefaultIndexManager] 1643701 issues reindexed in the background, in 29826927 millis.
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 ERROR abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All In Background FAILED. Indexer: DefaultIndexManager: paths: [/app/jira-home/caches/indexes/comments, /app/jira-home/caches/indexes/issues, /app/jira-home/caches/indexes/changes, /app/jira-home/caches/indexes/worklogs]
java.lang.NullPointerException
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:337)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:321)
at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:44)
at com.atlassian.jira.issue.util.IssueIdsIssueIterable.foreach(IssueIdsIssueIterable.java:27)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:320)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:174)
at com.atlassian.jira.index.IssueIndexHelper.fixupConcurrentlyIndexedIssues(IssueIndexHelper.java:117)
at com.atlassian.jira.issue.index.DefaultIndexManager.doBackgroundReindex(DefaultIndexManager.java:1095)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:322)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssuesInBackground(DefaultIndexManager.java:414) <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at com.sun.proxy.$Proxy15.reIndexIssuesInBackground(Unknown Source)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexIssuesInBackground(CompositeIndexLifecycleManager.java:123)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.doReindex(ReIndexBackgroundIndexerCommand.java:37)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.call(ReIndexBackgroundIndexerCommand.java:16)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254)
at java.lang.Thread.run(Thread.java:745)
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 INFO abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [action.admin.index.IndexAdminImpl] Re-indexing finished
xx.xx.xx.xxx,xx.xx.xx.xx 2018-06-09 17:18:43,893 JiraTaskExectionThread-508 ERROR abcde 541x10981514x3 116sphq /secure/admin/IndexReIndex.jspa [atlassian.jira.task.TaskManagerImpl] Task 'JIRA Indexing' failed.
java.lang.NullPointerException
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:337)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$3.consume(DefaultIssueIndexer.java:321)
at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:44)
at com.atlassian.jira.issue.util.IssueIdsIssueIterable.foreach(IssueIdsIssueIterable.java:27)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:320)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:174)
at com.atlassian.jira.index.IssueIndexHelper.fixupConcurrentlyIndexedIssues(IssueIndexHelper.java:117)
at com.atlassian.jira.issue.index.DefaultIndexManager.doBackgroundReindex(DefaultIndexManager.java:1095)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:322)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssuesInBackground(DefaultIndexManager.java:414) <+3> (NativeMethodAccessorImpl.java:62) (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at com.sun.proxy.$Proxy15.reIndexIssuesInBackground(Unknown Source)
at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexIssuesInBackground(CompositeIndexLifecycleManager.java:123)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.doReindex(ReIndexBackgroundIndexerCommand.java:37)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:59)
at com.atlassian.jira.web.action.admin.index.ReIndexBackgroundIndexerCommand.call(ReIndexBackgroundIndexerCommand.java:16)
at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:24)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:528)
at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:254)
at java.lang.Thread.run(Thread.java:745) 

 Kindly let me know the potential cause.

2 answers

@Manu Mishra  Try  the resolution  related to  jira-issue,  as i can  see from  the Log,  the error  abcde with a Null-Pointer Exception, so  it could be that  there is  a null issue num entry in the jiraissue table.

 

This could fix your problem.

https://confluence.atlassian.com/jirakb/indexing-fails-with-reindex-all-failed-error-429917142.html

Best!

@Moses Thomas I have already applied the fix mentioned in KB article. Though I forgot to mention in description. I am still facing the same error.

@Manu Mishra

Diagnosis

SELECT id, issuenum, project FROM  jiraissue WHERE  issuenum is NULL;

 Possible Root Cause

  • NULL value under the issuenum column of jirraissue table
  • Ticket registered to  atlassian  as a bug JSWSERVER-16473

Possible solution.

If the query from the diagnosis returns any results, run the following to remove the NULL entries:

DELETE FROM jiraissue WHERE issuenum IS NULL;

Have you tried this one from your data base server ? make sure you backup you Jira database when performing the delete query in-case.

Best

@Moses Thomas we always check the integrity before running re-index. However we still gets the error after validating integrity. I not sure what is conflicting.

@Manu MishraHave  you  tried what i have sent  to  you from  your  database server/SQL developer tool , if not contact  your database administrator to  try  resolution  i have mention above. this issue  is clearly related to solution mentioned.

@Moses Thomas If you are referring to the syntax. 

SELECT id, issuenum, project FROM  jiraissue WHERE  issuenum is NULL;

 I have already tried this fix but no luck. 

@Manu MishraDid you  get  any  result when you  tried this ?

if you  got the perform  this

DELETE FROM jiraissue WHERE issuenum IS NULL;

if  you  did not get any  result from  the Select query,

the issue could be something  else.

@Moses Thomas i did not get any values returned, therefore no use of running delete. 

backup the database and execute:

DELETE FROM jiraissue WHERE issuenum IS NULL;

in your db.manager.

@Danyal Iqbal how to perform delete operation in db.manger. please refer if any defined process is there. Also is this the only cause of Re-Indexing failure?

Nic Brough [Adaptavist] Community Champion Monday

"db.manager" means "whatever tool it is you would normally use to directly access the database to run SQL queries on it".

I would also seriously question why you are re-indexing every week - that, plus the fact you are getting errors smacks of something very wrong that you should be fixing instead of running regular indexing.

@Nic Brough [Adaptavist] we are running Re-Indexing every weekend because we have a very big instance and there are always some modifications happens. To synchronise the data we are required to run Re-Indexing every weekend when there is no load on server.

We are using SQL developer tool to access dB. Will this syntax run on that?

Also is there any pre-built script or service in Script Runner we can utilize to automate our regular weekend Re-Indexing?

Can you please elaborate more on "getting errors smacks of something very wrong that you should be fixing"

Nic Brough [Adaptavist] Community Champion Monday

If background re-indexing has that much of an effect on your server, you should be looking at the Data Centre option.  Also, if it is because of config changes, you should really be running a full re-index to capture all change.  In fact, a full re-index may fix the problem you are having (unlikely, but it could)

But going back to the orginal question, it looks like you have a null or damaged issue This should never happen, but can do in four broad cases.

  1. You've had a crash at just the wrong moment, and rubbish has been committed to the database
  2. You have run into a bug in Jira somewhere (not unlikely), one that is bad enough to commit rubbish to the database (very rare - while I'm not going to comment on Atlassian's record on bugs in general, I will say that their bugs that corrupt the database are very rare)
  3. You have 3rd party code that has written garbage to the database because of a bug, poor code or poor understanding of the database
  4. Someone has written data to the database without going through the Jira API
  5. (case 5 - a combination of the above - for example you can get perfectly good add-ons that write crud to the database on a crash)

Whatever the root cause, you should go find it and fix it. 

I would also not rule out "damaged index files", as that could be reading something it can't handle, then not writing it back, leaving a duff record that can't be fixed in background.  I would try a full re-index first to rule this out, as that would fix that sort of damage, and then wait to see if that cleans it out.

we always run full re-index. i hope you are referring to the "background re-index". We also run Lockdown but only once in a month. 

Nic Brough [Adaptavist] Community Champion Tuesday

No, you need to run a full re-index, not the background one.

But if you're already doing that regularly (which, again, you shouldn't need to), then you know that it's not fixing the problem.  Which means it's not a corrupt index and you really need to find out what is writing bad data.

I agree with Nic, a weekly reindex is not justified even if it improves the performance.

If background re-indexing has that much of an effect on your server, you should be looking at the Data Centre option.

I would say that your instance is undersized for the workload. While migrating to the data option is definitely recommended it might be possible to add resources to your server, optimize garbage collection and follow best practices for large instances to ensure a decent performance with the server version.

How many users do you have?

How big is your data(base) file? I believe you are using an oracle database.

Server specs?

If the database is not corrupt i.e none of the cases mentioned above, an oracle specialist should have a look at the database configuration, the alert/transaction log and the AWS report.

What does this SQL say:

alter index validate structure

@Danyal Iqbal Yes we are using Oracle, Yes we'r in planning to move cloud where we'll not be facing space issues. However thanks all of you for your inputs. 

Suggest an answer

Log in or Sign up to answer
How to earn badges on the Atlassian Community

How to earn badges on the Atlassian Community

Badges are a great way to show off community activity, whether you’re a newbie or a Champion.

Learn more
Community showcase
Published Thursday in Jira Service Desk

How the Telegram Integration for Jira helps Sergey's team take their support efficiency to the bank

...+ reading Fantasy). The same is true for him at the bank he works for: Efficiency is key when time literally equals money. Read on to learn how Sergey makes most of the time he has by...

205 views 0 2
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you