under Kubernetes env confluence cluster node won't join

[Alexander Tchaikovski]

Hello Colleagues. I've been trying to make a cluster out of K8S confluence Datacenter app. one node is ok, 2nd one won't start with strange error:

2022-05-20 12:39:57,932 ERROR [hz.confluence.priority-generic-operation.thread-0] [com.hazelcast.cluster] log [10.0.129.104]:5701 [wiki-confluence] [3.12.11] Node could not join cluster. A Configuration mismatch was detected: Incompatible joiners! expected: multicast, found: tcp-ip Node is going to shutdown now!

2022-05-20 12:39:57,932 WARN [hz.confluence.priority-generic-operation.thread-0] [com.hazelcast.instance.Node] log [10.0.129.104]:5701 [wiki-confluence] [3.12.11] Terminating forcefully...

2022-05-20 12:40:01,936 ERROR [Catalina-utility-1] [com.hazelcast.instance.Node] log [10.0.129.104]:5701 [wiki-confluence] [3.12.11] Could not join cluster. Shutting down now!

2022-05-20 12:40:02,006 WARN [Catalina-utility-1] [com.hazelcast.util.PhoneHome] log [10.0.129.104]:5701 [wiki-confluence] [3.12.11] Could not schedule phone home task! Most probably Hazelcast failed to start.

2022-05-20 12:40:02,028 ERROR [Catalina-utility-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below):

Node failed to start!

I believe there's something I missed to config but can't find what it is. Shared home is in place, data is ok, configured with multicast and didn't change anything else. 

[Yevhen]

@Alexander Tchaikovski it looks like K8s Hazelcast discovery does not happen. Is clustering turned on in values.yaml? Also, is this a new installation or you migrated an existing Confluence to Kube? Perhaps, there are leftovers of properties from a previous installation?

[Alexander Tchaikovski]

Hello @Yevhen . thank you for your comment. 

I gave up finally, installed STS latest version via Helm from scratch and migrated data using UI (backup/restore).

With these options SW installs smoothly, forms a cluster and works fine. Our luck we don't have much data so backup/restore is working ok, thinking of a case when we need to migrate 100Gb or more.. 

[Yevhen]

Perhaps the first time you migrated cluster properties which don't work in Kube. It's just my assumption as I don't have all the data to leap to a data-proven conclusion.

[Alexander Tchaikovski]

might be. we got it fixed, it's ok. 

there's another thing though. we've got some pics/thumbnails missing so tried to make reindex and propagate. but it throws an error: Failed to propagate.Can't access shared home... checked, shared-home is available on both nodes, users can read/write, free space is not an issue... tried to shut down 1 node, reindex, then up 2nd node and do it again - same error. is there any debug option or specific log file so we can use them? thank you!

[Yevhen]

@Alexander Tchaikovski I am afraid I can't suggest anything new - it's either disc space and/or permissions. Doesn't look like a k8s specific issue - just a general issue.

[Alexander Tchaikovski]

ok, got it. but in case it's a permission thing where can i read about the permissions? i suspect that shared-home has a wrong permissions. what the should be? because helm chart don't mention those, internet won't give much info on that. i mean at the moment root:root has all rights, they're drwxrwxrwx and i'm concerned. is it correct? if it's not how to set them up correctly? thank you.