Bitbucket Data Center on Kubernetes/Openshift without Multicast

Question

We would like to run a cluster of Bitbucket Data Center on Openshift. We can successfully run a cluster locally using Docker Compose with Multicast.

On Openshift, we cannot use Multicast and are therefore stuck with specifying a TCP/IP member list. As far as we understand the configuration, it is read on startup of a Bitbucket instance and when modifying it at runtime, the instances do not register the change. If we define the Pod IPs of two Bitbucket instances in the bitbucket.properties file, we can successfully run a cluster. However, this is not sustainable because Pod IPs will change eventually during restart of a container. We tried using Service IPs instead of Pod IPs, but without success.

We discovered that there is a Hazelcast discovery plugin for Kubernetes. It might be possible to integrate this into the Bitbucket Data Center.

Consequently, we have two questions:

* Has Atlassian planned to provide a dedicated Bitbucket Data Center version to run on Openshift without Multicast?
* Perhaps, we are not configuring Bitbucket Data Center correctly on Openshift. Are there instructions available for the setup?

Scott_nswp03 · Answer

Just further to this (TL;DR - still no success)...

I've since learned that GKE doesn't support multicast on it's own; but you can use a CNI add-on such as Weave Net to provide multicast support.

Unfortunately, even with Weave installed (and as far as I can tell, running correctly), Data Center nodes running on GKE are still unable to discover each other using multicast.

At this point I'm well over my head when it comes to CNI and container networking in general, so I haven't yet been able to determine where to go next.

The only hope I have is that in future Data Center versions, Atlassian adds support for Hazelcast's SPI discovery.

Scott_nswp03 · Answer

Hi Florian,

Unfortunately I don't have an answer to your question; I'm here because I have a similar problem with Confluence Data Center.

I've been trying to stand up a test/eval DC cluster on Google Kubernetes Engine (GKE), and found the same issue.

Using multicast, the nodes don't seem to discover each other (perhaps GKE doesn't support multicast either?)

Another option I tried was to use TCP/IP and supply the pod IP of the first node during setup. I was then able to successfully scale the stateful set to 2 replicas and everything worked fine.

However, if I then tried to scale to a third replica; the 2nd one immediately dies and restarts. When the 2nd one comes back, the 3rd dies and restarts...and this flapping continues indefinitely  (presumably because the 2nd/3rd nodes don't have their IPs listed in `confluence.cluster.peers`).

Regardless, I share your concern that hardcoding IPs is not ideal in an ephemeral environment like Kubernetes.

I know that the folks over at Praqma have a service that offers Data Centre on Kubernetes (https://www.praqma.com/services/ask/), so clearly it is possible...it just doesn't seem to work out of the box.

I too saw the Hazelcast discovery plugin you mentioned, but I also found this blog post that suggests Hazelcast discovery should just work on Kubernetes?

Would love to hear if anyone else has managed to get DC multicast discovery running in k8s.