8.13 keeps the Jira DC cluster healthy

 

Greetings, Community Members!

I am Ankit, a product manager for Jira DC. I would like to thank you for trusting us when we said that the Jira 8 platform upgrade will unlock massive benefits in future releases. We continually try to innovate and provide you the best product possible. I am excited to share a great new set of benefits in our most recent Long Term Support release, 8.13. Since the previous long term release we have made many architectural improvements to make the Jira DC cluster smarter and healthier. Read on to find out more about these improvements.

 

DBR (Document Based Replication):

The Jira DC cluster works smarter and the index remains consistent across the cluster

We’ve been improving the indexing process continually through the 8.x Jira release train. However, prior to Jira Data Center 8.13, each node of the cluster had to make separate calls to the database to redo the indexing work when something changed on another node. The more nodes, the more catching up was required by each node to get its index up to date. In heavy load situations, the index replication process could take up to 30 minutes depending on the number of custom fields, apps, and load. With Jira DC 8.13, our new architecture means that the index replication will not be delayed and the index will remain consistent across the cluster. Simply put, 8.13 delivers a more stable cluster and reliable consistency of data across each node in the cluster.

Apps and indexing time

We know that apps are critical to your work. If you have previously tried disabling certain apps due to index inconsistency problems in your Jira Data Center cluster, our new replication architecture can help you use those apps again.

Installed apps that work with the Lucene index may sometimes increase the issue indexing time which can lead to cluster nodes getting out of sync with their indexes.

8.13 mitigates index consistency problems even if apps take longer to index. Changes in the index are propagated faster and reliably between nodes streamlining collaboration.

Please read this if you are interested to know more about the test results and technical details related to DBR.

 

* Our tests used apps that introduced a synthetic delay in order to replicate real world conditions, but the actual results can vary based on apps in use.

 

Changes in architecture

Our previous architecture meant that the speed of index replication across the cluster was limited by the slowest node in the cluster due to single thread replay of all indexing operations. This is no longer the case as index changes on a node are now sent to other nodes directly. Therefore each node we add to the cluster increases the write capacity of the cluster.

Test results:

Our stress test on an 8-node cluster using Jira DC 8.5 shows that the index became highly inconsistent after the load of 200 requests per second.

Screenshot 2020-09-23 at 12.56.34 PM.png

 

Jira DC 8.13 was able to handle higher flat load of 330 requests per second with the index remaining consistent throughout. It proves that Jira DC can now handle the index consistency issues caused by high load.

Screenshot 2020-09-23 at 12.57.04 PM.png

Our tests used apps that introduced a synthetic delay in order to replicate real world conditions, but the actual results can vary based on apps in use.

 

Note: These architectural changes require each node to communicate directly with other nodes which can double the network traffic. When testing this on a 8 node cluster running a sustained stress load of 400 requests per second, the total network traffic volume amounted to 25% of total capacity of a 1Gbps link. Traffic will be lower if the load or cluster size is lower.

 

Custom Field Improvements:

In Jira Data Center 8.13, we’ve also improved custom field indexing. Jira field indexers will now be called only when the custom field is applicable based on scope/context of the issue and has a value assigned in the issue.

We ran tests with 7 million issues on a vanilla Jira DC instance which showed re-indexing time getting reduced by 50%. Our tests on a real-world instance with some of the popular 3rd party apps, showed an improvement of 20%. This improvement will be more visible on instances with custom fields having no value in large number of associated issues. This scenario is common in less used global custom fields. 3rd party apps that expose complex custom fields can significantly lower the performance benefits of these improvements.

With these improvements, Jira DC will mitigate some of the impact custom fields have on indexing time. Nevertheless, instance hygiene is still crucial, so it’s a good idea to run the custom field optimizer regularly to keep your setup in a good state.

 

These were some of the many improvements we have done to make Jira DC better. Read this to know more about what we’ve shipped since the last long term release & what to expect from the latest long term release.

3 comments

Dave Liao
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
November 25, 2020

Loving 8.13 so far, and looking forward to 8.14, especially those DVCS improvements...

Taranjeet Singh
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
January 11, 2021

These are great improvements! Jira DC 8.13 looks much more promising due to these improved features, especially related to indexing replication speed, and custom field indexing.

Gonchik Tsymzhitov
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
May 30, 2021

@Ankit Tiwari  Thank for improvements, do you have a plan to share the some tools to checking index status?

Because errors like this too surprised to me

NodeReindexServiceThread:thread-0 INFO [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered=true, notRunningCounter=412, paused=true, lastPausedStacktrace=java.lang.Throwable

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events