We make a lot of configuration changes throughout the week to support our customers. As such we decided to get on a cadence of weekly weekend maintenance reindexes, and restarts. We utilize a spare node in our cluster to handle the reindex and once that completes we perform rolling restarts of all other nodes in the cluster.
Sounds great, but we consistently have problems with index integrity. Things like filters and dashboards missing from indexes. To resolve we have to manually copy index from our spare node to the affected nodes.
Does anyone else experience this? Anyone have a better way of handling this that eliminates this issue?