JIRA 8.13 indexing problem

[Saleh Houshangi]

Recently we have updated our JIRA from version 7.9 to 8.13 , All thing were good until we had to re index JIRA , our architecture is base on three node in cluster mode . after full re index jira( we chose this way because our JIRA server provides thousand issues per day). .Although we haven't seen any error during and after re index , tomorrow after jira server were under the load we have seen some strange error and later on cpu has been used lots of by jira also memory has been used full and got use swap also some time it get full gc problem we monitored logs and those logs indicated we have problem in indexing document ,subsequently we decided to full re-index again but despite indexing has completeed successfully we saw some error implies that in re index process skipped from some document:

Structure-Jobs60b18591 Queue-Thread#137213 WARN anonymous     [c.a.j.structure.cluster.ExclusiveClusterJob] Full reindex is in progress (Structure cannot lock JIRA against full reindex) or index is inconsistent with the database. "ApplyJob" is skipped this time

Here is some logs have been appeared as below when jira server starting to work after reindex :

2021-06-22 10:53:36,092+0430 Structure-Jobs60b18591 Generator-Thread#136762 INFO a.sharifi 595x47427692x2 1jlsbdh 10.20.xx.xx,10.xxx.0.xx8 /rest/structure/2.0/poll [c.a.
jira.index.MonitoringIndexWriter] [lucene-stats] flush stats: snapshotCount=298, totalCount=233028, periodSec=302, flushIntervalMillis=1016, snapshotNoDocs=0, totalNoDo
cs=0, indexDirectory=/opt/atlassian/application-data/jira/caches/indexesV1/issues, indexWriterId=com.atlassian.jira.index.MonitoringIndexWriter@3fbf80fa, indexDirectory
Id=MMapDirectory@/opt/atlassian/application-data/jira/caches/indexesV1/issues lockFactory=org.apache.lucene.store.NativeFSLockFactory@1665dd9e
--------------------------------------------------------------------------------------------------------------------2021-06-22 11:43:17,594+0430 Structure-Jobs60b18591 Generator-Thread#136758 WARN s--aleki 595x47426680x2 1isjx1g 10.xx0.151.xx,10.1xx.0.xx /rest/structure/2.0/poll [c.a
.j.s.forest.gfs.AbstractTransformingForestSource] SecuredForestSource(a660e59b)(GeneratingForestSource(eea658af)(null)({"structureId":10965})) cannot acquire lock in 30
000ms

[David Niro]

Hello @Saleh Houshangi

David from ALM Works here.  We will be happy to help review the logs and work with you to identify the cause of your re-index error.  

To facilitate this, it will be best to work directly through our support channel.   Please send us a quick email to support@almworks.com. 

I look forward to working with you!

Best Regards,
David Niro
www.almworks.com

[Niranjan]

@Saleh Houshangi - Structure is a heavy plugin - consumes more system resources and can cause performance issues. I would recommend that you

1. Disable structure for a day and monitor the performance.
2. Take a node out of your load balancer and reindex it.
3. After completion of indexing on one node, copy the indexes to other nodes.
4. Analyse GC logs and work with the plugin vendor (Structure) and make changes as requried. You may need to follow the best practices that the plugin vendor may recommend.