Why does each node in Jira data center have its own copy of lucene indexes

Paul
Contributor
August 3, 2017

From reading this article on how Jira indexing works in a data center environment; each node has its own copy of the index. 

Reading the Elastic Search documentation on how it functions in a cluster; there is only one instance of an index which may be split across each node.

My question is why does each Jira node have its own copy of the index? It isnt efficient and could mean getting different results dependent on which node you are on.

3 answers

1 accepted

1 vote
Answer accepted
Nic Brough -Adaptavist-
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
August 3, 2017

It's actually highly efficient, in the one single area that matters for an index file: speed.

Shared file systems are simply too slow to serve up index files at the rate even a lightly used JIRA system needs it to. 

Paul
Contributor
August 7, 2017

Thanks for the reply Nic,

In our multi node data center environment we have a cron job that triggers an overnight background re-index on an admin node. Once the index is built this gets copied across to the other nodes. As events are generated these are broadcast to the other Jira nodes. 

Every night we trigger the background re-index  on the admin node to avoid the risk of indexes becoming out of date.

Does this setup sound correct?

Nic Brough -Adaptavist-
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
August 7, 2017

Not really.  There's generally no need for regular re-indexing.  I'd take a look at why you're doing this regular indexing and see if it's actually useful.

2 votes
Paul
Contributor
December 6, 2017

We are suffering with ongoing issues with our Jira data centre nodes having corrupted indexes.

Atlassian havent got this right at all. For data centre environments using netapp or other shared storage I dont agree with you about the speed issue and it being faster having indexes locally.

A more scaleable robust solution would be where each node contributes towards building  an index in ElasticSearch (which can itself be multi node and resilient).

Splitting the reindex task across nodes would also reduce the reindex time from the several hours it takes currently.

Nic Brough -Adaptavist-
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
December 6, 2017

The fastest shared storage I've measured is 10 times slower than equivalent local storage.  A shared elastic search would be nice, but it would still introduce far too much delay to be useful.

0 votes
Kav
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 15, 2018

Paul, we cannot used a Shared storage system as Nic pointed out for the Lucene indexes due to the access latency of non-local storage. In fact we highly recommend SSD devices for local index storage as well for the best reindex/search performance

Your problem sounds like the JIRA DC bug we have run into (among the various ones) in 7.2.12 and lower. And 7.5.2 as well that you are using. I saw another thread from you about JDC Node hanging and I highly recommend upgrading to 7.6.4 or higher (Enterprise Release). We have seen considerable instability with JDC in 7.4/7.5 and most of the issues found have been  addressed in 7.6.x. Besides staying on an enterprise release will get you significant advantages detailed here

Also you shouldn't have to sync indexes. I recommend if out-of-sync indexes ever becomes a concern to pull the node out of the cluster (out of the LB) and do a foreground reindex and put the node back into the cluster after verifying that the indexing was successful. The indexes among the nodes will be synced automatically and there shouldn't be any need for you to copy indexes over to each of the node. In fact I recommend you don't do this because you might lose any indexes built in the interim. Please see the KB on "Reindexing JIRA Datacenter without downtime"

Paul
Contributor
April 15, 2018

Hi Kavitha,

Many thanks for your reply.

If a search returns results in a few seconds then I'm happy and for that to happen it doesn't mean indexing and search have to be sat on the same box as Jira; that doesnt scale plus you could get inconsistent resultants dependent on which box you hit.

Of the large scale enterprise systems I have worked on they have an Elastic cluster with application servers carving up the task of indexing, feeding the Elastic cluster with Elastic serving out search results. These are large scale systems using something like a netapp metro cluster for storage. I hear what you and Nic have said about local storage however good design of your Elastic Search cluster can mitigate this.

Jira indexing only runs on one node, so in our 4 node Jira data centre instance, indexing is taking 4 times longer than if it had been designed to allow all 4 nodes to participate in indexing.  In fact its longer than that as it has to copy the indexes to the other 3 nodes after the index is built.

 

 

 

mario truber October 10, 2023

see this answers about jira arquitecture not supports a simple index centralization of a atlassian team means ridiculous, a arquiteture largest difused and used for any application arround the globe. let the user make your proper choice, use split index or centralized index, let the user improve your infraestructure, if the response looks nice or note is this a user choice, not a legacy arquitecture from application(jira).

Nic Brough -Adaptavist-
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
October 10, 2023

It is not ridiculous to build something that works.

The problem that single-node indexing helps with is not with the software or even the architecture.  It's down to physics (i.e. reality). 

Remote storage, which is what you would have to use to do split or centralised indexing, is slow.  Jira needs the index to be fast, so it has been built to use the fastest storage it can - a disk local to the node.  Even if two nodes are physically next to each other, equipped with fast hardware and a fast link between them, their storage is going to be at least 1/5th the speed when accessed by the other node.

No matter how you cut it, you can't argue with the physics.  

Part of this conversation was about Elastic search, which is very good at distributed search, but in multi-node solutions, under the hood, when configured to work fast on multiple nodes, is still effectively working with a local cache.  (Personally, I'd prefer Elastic to be the indexing in Jira, it's nicer to work with than Lucene, and I suspect it would be better for Jira's indexing structure as well, but it would not be any faster than the single-node indices Jira has on DC)

Like mario truber likes this

Suggest an answer

Log in or Sign up to answer