nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why does each node in Jira data center have its own copy of lucene indexes

Paul_28363u

From reading this article on how Jira indexing works in a data center environment; each node has its own copy of the index.

Reading the Elastic Search documentation on how it functions in a cluster; there is only one instance of an index which may be split across each node.

My question is why does each Jira node have its own copy of the index? It isnt efficient and could mean getting different results dependent on which node you are on.

Accepted answers

Nic Brough -Adaptavist-

It's actually highly efficient, in the one single area that matters for an index file: speed.

Shared file systems are simply too slow to serve up index files at the rate even a lightly used JIRA system needs it to.

All comments

Nic Brough -Adaptavist-

It's actually highly efficient, in the one single area that matters for an index file: speed.

Shared file systems are simply too slow to serve up index files at the rate even a lightly used JIRA system needs it to.

Paul_28363u

Thanks for the reply Nic,

In our multi node data center environment we have a cron job that triggers an overnight background re-index on an admin node. Once the index is built this gets copied across to the other nodes. As events are generated these are broadcast to the other Jira nodes.

Every night we trigger the background re-index on the admin node to avoid the risk of indexes becoming out of date.

Does this setup sound correct?

Nic Brough -Adaptavist-

Not really. There's generally no need for regular re-indexing. I'd take a look at why you're doing this regular indexing and see if it's actually useful.

Paul_28363u

We are suffering with ongoing issues with our Jira data centre nodes having corrupted indexes.

Atlassian havent got this right at all. For data centre environments using netapp or other shared storage I dont agree with you about the speed issue and it being faster having indexes locally.

A more scaleable robust solution would be where each node contributes towards building an index in ElasticSearch (which can itself be multi node and resilient).

Splitting the reindex task across nodes would also reduce the reindex time from the several hours it takes currently.

Nic Brough -Adaptavist-

The fastest shared storage I've measured is 10 times slower than equivalent local storage. A shared elastic search would be nice, but it would still introduce far too much delay to be useful.

Kav

Paul, we cannot used a Shared storage system as Nic pointed out for the Lucene indexes due to the access latency of non-local storage. In fact we highly recommend SSD devices for local index storage as well for the best reindex/search performance

Your problem sounds like the JIRA DC bug we have run into (among the various ones) in 7.2.12 and lower. And 7.5.2 as well that you are using. I saw another thread from you about JDC Node hanging and I highly recommend upgrading to 7.6.4 or higher (Enterprise Release). We have seen considerable instability with JDC in 7.4/7.5 and most of the issues found have been addressed in 7.6.x. Besides staying on an enterprise release will get you significant advantages detailed here

Also you shouldn't have to sync indexes. I recommend if out-of-sync indexes ever becomes a concern to pull the node out of the cluster (out of the LB) and do a foreground reindex and put the node back into the cluster after verifying that the indexing was successful. The indexes among the nodes will be synced automatically and there shouldn't be any need for you to copy indexes over to each of the node. In fact I recommend you don't do this because you might lose any indexes built in the interim. Please see the KB on "Reindexing JIRA Datacenter without downtime"

Paul_28363u

Hi Kavitha,

Many thanks for your reply.

If a search returns results in a few seconds then I'm happy and for that to happen it doesn't mean indexing and search have to be sat on the same box as Jira; that doesnt scale plus you could get inconsistent resultants dependent on which box you hit.

Of the large scale enterprise systems I have worked on they have an Elastic cluster with application servers carving up the task of indexing, feeding the Elastic cluster with Elastic serving out search results. These are large scale systems using something like a netapp metro cluster for storage. I hear what you and Nic have said about local storage however good design of your Elastic Search cluster can mitigate this.

Jira indexing only runs on one node, so in our 4 node Jira data centre instance, indexing is taking 4 times longer than if it had been designed to allow all 4 nodes to participate in indexing. In fact its longer than that as it has to copy the indexes to the other 3 nodes after the index is built.

mario truber

see this answers about jira arquitecture not supports a simple index centralization of a atlassian team means ridiculous, a arquiteture largest difused and used for any application arround the globe. let the user make your proper choice, use split index or centralized index, let the user improve your infraestructure, if the response looks nice or note is this a user choice, not a legacy arquitecture from application(jira).

Nic Brough -Adaptavist-

It is not ridiculous to build something that works.

The problem that single-node indexing helps with is not with the software or even the architecture. It's down to physics (i.e. reality).

Remote storage, which is what you would have to use to do split or centralised indexing, is slow. Jira needs the index to be fast, so it has been built to use the fastest storage it can - a disk local to the node. Even if two nodes are physically next to each other, equipped with fast hardware and a fast link between them, their storage is going to be at least 1/5th the speed when accessed by the other node.

No matter how you cut it, you can't argue with the physics.

Part of this conversation was about Elastic search, which is very good at distributed search, but in multi-node solutions, under the hood, when configured to work fast on multiple nodes, is still effectively working with a local cache. (Personally, I'd prefer Elastic to be the indexing in Jira, it's nicer to work with than Lucene, and I suspect it would be better for Jira's indexing structure as well, but it would not be any faster than the single-node indices Jira has on DC)

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

jira

data-center

lucene