Recently we have had a number of indexing failures on our 4 node Jira data centre installation. Indexes were corrupt and restoring from backup indexes was also failing. Our system administrator is currently holding his head in his hands wondering how to fix it.
Compared to Bitbucket where ElasticSearch was added to the product it looks like Jira Data Centre still has the old single node Search architecture and it doesnt work well in Jira Data Centre.
Each Jira node looks to have its own embedded lucene instance with indexes being built on a node and copied across to the other nodes. Each node has its own copy of the indexes. Our recent problems with corrupted indexes has prompted me to write this post and suggest an alternate approach.
Where Jira is
ElasticSearch is embedded in Jira, each node has it's own copy of the complete index. This doesn't scale, isnt efficent or performant and from recent experience isn't reliable.
Where Jira should be
ElasticSearch should be a separate install preferably on a separate server or servers. Each node contributes to generating indexes that are used by all nodes (a single source of truth). Indexing times are reduced as the task can be split across all nodes. Indexing is only ever a background task.
