Kubernetes Azure - Error during setup

[Richard Adler]

During the setup of confluence data center in a kubernetes cluster in azure via the helm chart, I get the following error :

Caused by: com.atlassian.secrets.api.SecretServiceException: Error applying read only permissions to file   /var/atlassian/application-data/shared-home/keys/ATL_SECRET_AES_

This problem is occurring in the confluence version 9.3.1. When I configure the version 9.0.2, the error is not happening. 

My persistent volume is referencing to an Azure file share. The current config of the persistent volume is:

apiVersion: v1

kind: PersistentVolume

metadata:

  annotations:

    pv.kubernetes.io/provisioned-by: file.csi.azure.com

  name:  confluencemax-shared-vol-pv

  namespace: confluencemax

spec:

  capacity:

    storage: 1Gi

  volumeMode: Filesystem

  accessModes:

    - ReadWriteMany

  persistentVolumeReclaimPolicy: Retain

  storageClassName: azurefile-csi

  csi:

    driver: file.csi.azure.com

    volumeHandle: "Ressourcegroupname#storagename#sharename"  # make sure this volumeid is unique for every identical share in the cluster

    volumeAttributes:

      #resourceGroup: resourceGroupName  # optional, only set this when storage account is not in the same resource group as node

      shareName: sharename

    nodeStageSecretRef:

      name: azure-secret-sharednode

      namespace: confluencemax

The ATL_SECRET_ secret file is created in the Azure file share and has following permissions: 

-rwxrwxrwx 1 root confluence 32 Feb 24 09:07 ATL_SECRET_AES_

Also after entering manuell the the command chmod a=r ATL_SECRET_AES the permissions are changed. Has anybody a clue how to allow file permissions changes for the confluence user

[Yevhen · Accepted solution]

@Richard Adler do you mind sharing the entire stacktrace?

[Dylan Rathbone]

@Richard Adler can you please also confirm if the Confluence system is actually available and can be used having hit this problem?

[Richard Adler]

The confluence system will not start with the described error.

Anyway I found a solution to this problem. It seems like Azure file share in the background don't allow the change of the ownership for mounted files, see https://learn.microsoft.com/en-us/answers/questions/1693979/unable-to-change-ownership-of-mounted-partition-of.

The ATL_SECRET_AES encrpytion is introduced in version 9.1, that's the reason why it's only occurring in the new versions and not in 9.0

In the end I moved from Azure files to Azure blob storage and it worked. But you have to use the nfs protocoll, with blobfuse there are other problems. 

PS: When someone in the future want to use Azure files anyways, try to configure the mount options in the pv and change the uid and guid to 2002, which reference the confluence user created by the helm charts. I didn't pursue this approach any further, because of a Java Core Exception. But this error also occurred later with the Azure blob storage and let me investigate it a little bit more. It seems related to the following post https://github.com/bcgit/bc-java/issues/1999. In the end I changed the used vm scale set to another with an other cpu architecture and the problem with the Java Core was gone. 

[Yevhen]

@Richard Adler thanks for the update. We'll fix this in a new version. This should not be a fatal error so we just log it instead of throwing an exception.