Lucene Index Directory over NFS

[Confluence]

Hello,

Our company has an instance of Confluence Server under Kubernetes and has the home directory confluence in NFS and it is not possible to change it because the entire Kubernetes infrastructure works on NFS.

My question is this: What is the impact of working under NFS apart from performance problems? We are evaluating changing the product to Confluence Datacenter but we will have the same NFS problem so I need more information about the problems of working with indexing under NFS.

A greeting.

[Alexis Robert · Accepted solution]

Hi @Confluence , 

having the indexes on NFS shares is usually not recommended because of performance issues. The documentation from Atlassian clearly states it:

Using Lucene over a NFS mount will cause stability issues with the instance. Some of the more common symptoms are slow search speeds (as NFS mounts are typically slower than local) and running out of open files as detailed in the below KBs.

If you believe your NFS infrastructure is performant enough to handle lucene indexes, then you can always use it. You should however be aware of the risks involved.

Let me know if you have any questions, 

--Alexis

[Confluence]

Hi @Alexis Robert ,

So the problems are at the level of operations on disk I/O and does not involve the loss of information. I am right?

In the case that we are on an NFS inside an SSD where the I/O is relatively better, we would not notice the slowness?

I do not understand why the Liferay application that also uses the indexed by Lucene does not have this problem with the NFS

Regards.

[Nic Brough -Adaptavist-]

No, it's not just to do with speed, although that is a factor.  If you've provisioned a fast disk with NFS over it, it reduces the risk and gives you better performance, but you still should move to a supported file system.

There is a very low risk of data loss.  But a much greater risk of performance issues and down-time. I almost all cases, the worst case is the index becomes totally corrupt and that can easily be fixed by running a complete re-index of the system.  So when you do run into a problem, the impact is "downtime while we re-index".  But there is a small chance of bad data going back into the database as well.  You will find creeping index corruption as well.

You are also totally unsupported if you remain on NFS.  Any support call will quickly lead to "move to a supported file system, and try it again"

[dominik_manser]

Hi

Has there be any development around this issue? We also use Confluence on Kubernetes and it is a big problem to find an option for the index mount as everything works over NFS. 

Is something planned for supporting NFS or any recommended way to go from here?

Regards,

Dominik 

[Nic Brough -Adaptavist-]

Not on the Atlassian side.  There's nothing they can do - the problems are with NFS and Lucence, not what Atlassian stuff is doing with it.

They only thing Atlassian could do on their side would be to swap to another indexing system.  I do not know if there are plans to do that.

[dominik_manser]

Okay yeah I thought as much. But maybe in order to get ready so that customers can deploy Atlassian-software in a cloud-native way that could be a priority. 

What happens to a cluster if I delete a node and the index of that node is lost? The node would go offline and if a new node joins the cluster it will sync again with the running nodes, is that correct? 

What happens if I delete all nodes and the index is completely lost? Can it be rebuilt on startup?

[Nic Brough -Adaptavist-]

They already do that, just not for cloud systems they can't support.  See the ami files for AWS as an example.

Yes, dropping a node loses that node, and a new node will synchronise.

If you completely destroy the index, then you will get a lot of errors, but hitting admin -> index -> rebuild will recrate a new set.

[dominik_manser]

Thanks for your answers :)

I've seen the support for AWS with templates etc. Unfortunately I'm not free to deploy to AWS at the moment.

[Steve Letch]

Just thought I'd add, that now the index recovery is in, when you start up the pod the index will recover from the shared home snapshot, however an issue I'm noticing is that even though it appears restored, the popular content tab is only showing the popular questions and not the popular pages, Am speaking to support out it for a workaround if any.

[Nic Brough -Adaptavist-]

Yep, your index is damaged, the "workaround" is "do not put your index on an unsuitable file system".  Move it to a fast local disk.

[Steve Letch]

As said, its on fast local storage, but local storage gets wiped on pod rest

art. unfortunately our kub system isnt built on Block but Vast, maybe one day 

[Nic Brough -Adaptavist-]

You need to move the storage to a fast local perstistent disk, and stop wiping it with your "pod restart".