Yevhen Atlassian Team

@Mark Hubers of course it is possible to scale Bitbucket StatefulSet to as many replicas as you need. What issues with a PVC do you face?

@Ashish Thakur perhaps you will find official docs helpful https://atlassian.github.io/data-center-helm-charts/platforms/openshift/OPENSHIFT/

@Haquiel Jusino make sure you set https://github.com/atlassian/data-center-helm-charts/blob/main/src/main/charts/jira/values.yaml#L753 and https://github.com/atlassian/data-center-helm-charts/blob/main/src/main/charts/jira/values.yaml#L789 to true. This way, configmaps will be created and mounted as files to the container.

I see. Access to others have been revoked on purpose, following recent DC CVEs. Is it an option for your to make nfs dirs writable for respective container users and run containers with the default securityContext? Also, it'd be great if you can share your existing securityContext. I think it's runAsGroup that makes things…

@Daniele Cortis Permissions for install dir have indeed been updated. However, It looks like you start containers as some different user. Jira (like Confluence and Bitbucket) user has read and execute permissions for the install directory (550). Can you share the securityContext that you use? Also, I expected to find the…

@Valentina Mannara sorry for a very delayed response. You should follow sizing guides for the conventional installations and make K8s requests similar to EC2 instance sizes, i.e. if you had VMs with 4 cores and 16Gb memory, you should aim at having the same pods. 

@Bryan Guffey yes, Jira 9.1.x is known to have a much more robost mechanism of restoring cache which is especially important when nodes start.

This is to emphasize that leaving it as emptyDir: {} will result in data loss. You can definitely use a custom volume for local home if you run in a non clustered mode.

In a non clustered mode, confluence home is the source of truth. So, it should be EBS backed.

@robert.haskins the sidecar should mount local-home volume mount that does not have to be NFS. It can be RWO. And then it's just all about flient bit configuration which files to tail and where to send them

/var/atlassian/application-data/confluence is a local home volume. If you scale your statefulset pods will overwrite it. I think the best way is to have a sidecar container that tails selected logs and sends them to logging backend.

@Thomas Langhorne as much as I want to help, it looks like the forum format is no longer suitable to troubleshoot the issue. I suggest opening a ticket with support. To me it looks like hazelcast-networking issue. We never tested in BigBang and/or istio. Maybe the second node is claiming to be a master for some reason.

@Thomas Langhorne the logs you have posted are from pod-0 or pod-1? Were you able to get a 1 node cluster?

@Thomas Langhorne I think this is what you need to do: 1. Scale to 0 2. Add database values to values.yaml (IMPORTANT!). 3. Run helm install again. What happens is that you finish the installation manually, then scale up, Confluence container generates confluence.cfg.xml based on available env vars, but Confluence thinks…

@rughvi8784 this init container has been added to address permissions issue that you `might` have with nfs PVs. You can disable it, and if products can run without fatal errors (caused by inability to write to shared home), then you're good to go.

@Thomas Langhorne it'd be great if you can share the complete atlassian-confluence.log for the 2nd node.

@Bryan Guffey let me clarify things. 1. You can scale Jira (like any DC product except Bamboo) to more than 1 replica. You can't scale more than 1 at a time but this is the default behavior of K8s StatefulSet. So, if you scale to 3, you will see pod-1 first, and pod-2 will be created only after pod-1 passes its readiness…

This does not look like a fatal error, and I am pretty sure you can ignore it.

Unfortunately, securityContext for init containers can't be set. However, nfs-permission fixer init container can be disabled (it fixes permissions for shared home volume, and it's not always necessary)

This container runs as root, maybe it makes your admission controller unhappy.

I'm sorry - helm upgrade

Correct, none of them will. All you need to do is to set confluence.clustering to true, Helm chart will do the rest. You said you migrated your local-home and I think that's the problem. Can you please 1. Scale down to 0 2. set: ``` confluence: additionalEnvironmentVariables: - name: ATL_FORCE_CFG_UPDATE value: "true" ```…

@rughvi8784 Just keep adding missing restrictions to securityContext. Btw, the admission controller may not like init containers as well. Do you have nfs fixer init container enabled in volumes.sharedHome? You can try setting securityContext for the entire pod (confluence.securityContext) You need to figure out the missing…

I am not familiar with IronBank Helm Charts. I do recommend using the official Helm charts (you can still use IronBank Confluence image though)

Make sure you add runAsUser: 2002 to securityContext You will also want to set https://github.com/atlassian/data-center-helm-charts/blob/main/src/main/charts/confluence/values.yaml#L834 to true if you are running as non root.

try adding fsGroup: 2002 to containerSecurityContext to allow Confluence user to write to local home.

@Kenneth Epker did you enable clustering in values.yaml? >Configured first node for clustering How did you do that?

@rughvi8784 can you share the exact errors from pod events when you add containerSecurityContext?

The deployments are continuously tested. The default values are close to prod values, and having 2 vCPU available isn't a huge resources request, imho.

@AM perhaps your nodes do not have enough cpu or memory. You may want to override the default requests Adding more nodes won't help if they are the same nodes with less than 2 vCPU available. You may need to use larger nodes or decrease cpu request.