nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Cluster Index Replication Health Check Fails during reindexing

Shrikant Bijapurkar _NTT DATA_

What is the reason for this error?

Reindexing Completed.png

Accepted answers

All comments

Ravina

Hi @Shrikant Bijapurkar _NTT DATA_ Technically this health check error will be shown when full indexing is running or just completed on the node.

In full indexing the current index is deleted and new index is build from the scratch, while the indexing is running on the node there is a Jira internal service NodeReindexService is keep running every 5 sec to distribute index change from one node to another so that indexes will be upto date on all other nodes across the Jira, this process is resource intensive and sometime it takes time to sync the indexes between the nodes and index health check gets timeout and we will see this error message.

Once the indexing is completed you can copy that updated index to the other nodes so that index on the other nodes will be updated this latest index to have the updated data, this process will take time based on the instance and index size but typically it's a fast process and once it is completed error will be gone in the next health check, this is common issues we face when the full indexing is triggered or completed.

Capture 2024-07-08 at 10.49.23.png

If the error not auto-resolved after hour or so then there are various reasons for this that need to check as mentioned in the below KB document that you can check,

But before checking the KB document I would suggest understand how to indexing is work in Jira Datacenter so that it will hep you understand the issue and find the cause of this error.

https://confluence.atlassian.com/jirakb/how-indexing-works-in-jira-1167744587.html

https://confluence.atlassian.com/adminjiraserver/search-indexing-938847710.html

Related KB documents for the Cluster index replication errors.

https://confluence.atlassian.com/jirakb/cluster-index-replication-health-check-fails-in-jira-data-center-738722192.html

https://confluence.atlassian.com/jirakb/cluster-index-replication-health-check-fails-in-jira-data-center-due-to-jira-charting-plugin-974387065.html

https://confluence.atlassian.com/jirakb/cluster-index-replication-health-check-reports-delay-when-the-database-is-in-a-different-timezone-than-jira-1051984214.html

If you Datacenter is setup correctly then 90% of time this error will be auto-resolved once the updated index is copied to other active nodes and next health check passed in specified time.

Let me know in case of any questions.

Shrikant Bijapurkar _NTT DATA_

Thanks a lot for your answer, Ravina.

Can you pls elaborate what node details to put in "From" and "To" boxes?

Our is a 3 node cluster.

Also, Jira spawns additional nodes if the CPU usage goes above threshold during full reindexing.

So, I am not sure which node number to put in "From" and "To" boxes.

Guidance around this would be helpful.

All I know is the node number that goes down during full reindexing.

Shrikant Bijapurkar _NTT DATA_

Attached picture shows our 3 nodes.

1 of these goes down when the full reindexing is triggered.

Screenshot 2024-07-08 at 12.19.08 PM.png

Ravina

@Shrikant Bijapurkar _NTT DATA_Nodes does not goes down as you can see the uptime of all the three nodes are same if node goes down then the uptime of the node will be reset, but yes, when the full indexing is triggered node does not accept the active traffic until the full indexing is completed, so it is recommended to offload/remove the node where indexing is triggered from the load balancer so that user traffic will be redirected to the other active nodes.

Regarding the index copy "FROM" to "TO" node, once the indexing is completed on the triggered node (In your case it is the node that goes down/not accessible to the users) so this will be your FROM node, you can also identify these nodes based on the node-id as nodes will have a unique id's identify then, so for example in your case if you started/trigger the full indexing on node - i-078239b... and completed the indexing then this will be your FROM indexing and node - i-02b7afca... will be your TO node in first case, click on copy index button, wait for sometime then perform the same action to copy the index to another node keep FROM node as same and use node - i-0368df0... in TO and click on copy index.

As you have setup a auto scaling to scale another node if the CPU load increase then in that case as this will be a new node joining the cluster once the node is started then the NodeReindexService will look for the latest copy of the index from other nodes or shared home and build the index for the new node joining the cluster, so in case of new node is getting scale technically there is no need to perform this copy index action.

So best practice to follow the full indexing are:

Always perform the full indexing on the low traffic hours or weekend
Perform the regular full indexing on weekly basis for the large instance or 15 days to per month for moderate to small instance to keep the indexes upto date as Jira search results are depends the index health/update.
In Jira Datacenter instances remove the node from the cluster/load balancer when indexing is running so that user traffic will not go that node and redirected to other active nodes where indexing is not running and add the node back to the cluster/load balancer once the indexing is completed and copy that latest index from the node where indexing is trigger to the other nodes where indexing not triggered.
Automate this indexing, adding/removing node from the cluster/load balancer and scheduled it to perform on the weekend so that there does not require manual intervention and indexes will be updated on weekly basis.
Follow this Atlassian KB document to increase the indexing speed by adding the mentioned parameters in the jira-config.properties file that need to created in the Jira local home if not already present.

Let me know in case of any question.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

jira

jira-data-center

data-center