ScriptRunner low performance on Multi Node

[Eloi Serret]

I'm deploying Jira DC in AKS environment. The fact is that my ScriptRunner Scripts and functions (like copyProject) is running ok when I deploy only one node - CopyProject takes about 4 mins - but when I deploy more than one node, the scripts take so long time to finish - CopyProject takes more than 30 mins - .

I suspect that the problem is related to Cache Syncro but I'm note sure. For sure the resources are not a problem: Memory and CPU is OK, DataBase is Ok, the FileShare is from Premium Storage Account...

Please help me!

[Jim Knepley - ReleaseTEAM · Accepted solution]

You're right, the Jira index is synchronized across nodes when copying a project. The index lives on the individual nodes, so your node disk performance (specifically IOPS) will make a big difference.

Depending on which disk type you're using now, you might try using Azure ultra disks to see if that helps, then scale back to find the right balance of price and performance.

[Eloi Serret]

Hi @Jim Knepley - ReleaseTEAM !
Thank you for your reply! I'll try this!! In any case I can specify that now I'm using a Premium Storage Account in Azure Files. My Local homes are automatically created by Helm (SMB File Shares). For now these are small disks, and have small IOPS (3000), but I tried increasing it to the max (7200) and it has no result..

On the other hand I have another Premium Storage Account where I have the Shared Home in an NFS FileShare of 3TB.

I understand that, in this case, the important Storage Account is that holds Local Homes so I would try ultra disks in this way, right?

Best regards,
Eloi.

[Jim Knepley - ReleaseTEAM]

Housing the index in Azure premium storage with 3000 IOPS should be fine.

I'm a little curious about your comment that local homes are SMB shares. Jira's home is on an SMB mount backed by Azure Files, I can't imagine that will perform well.

[Eloi Serret]

Hmmm what you mean? I followed the Atlassian Guide as I could..

What storage class or pv options would you recomend for Jira? I'm open to any idea..

Best regards and thank you so much.
Eloi.

[Jim Knepley - ReleaseTEAM]

Jira stores Lucene indexes in the Jira home directory. If Jira was installed using the Windows installer, the default location of the Jira home directory is:

C:\Program Files\Atlassian\Application Data\JIRA

If that's where Jira is installed, you should be fine.

If Jira's installed somewhere else, particularly in a local home, then SMB overhead could have a serious impact on performance.

[Eloi Serret]

Hi Jim,

nothing about Windows installer.. I deployed this Jira through Helm chart in an AKS environment. So when I mean Local-homes, I'm talking about the local-home of each node. This local-homes are hosted in Azure Storage Account Premium, and this FIleShares are SMB type https://learn.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share?tabs=azure-portal

When AKS raise a Jira pod, it creates a new FileShare in this StorageAccount and mount it to the pod.

This is the Values:

  localHome:

    # Dynamic provisioning of local-home using the K8s Storage Classes

    #

    # https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic

    # https://atlassian.github.io/data-center-helm-charts/examples/storage/aws/LOCAL_STORAGE/

    #

persistentVolumeClaim:

      # -- If 'true', then a 'PersistentVolume' and 'PersistentVolumeClaim' will be dynamically

      # created for each pod based on the 'StorageClassName' supplied below.

      #

      create: true

      # -- Specify the name of the 'StorageClass' that should be used for the local-home

      # volume claim.

      #

      storageClassName: azurefile-csi-premium

      # -- Specifies the standard K8s resource requests and/or limits for the local-home

      # volume claims.

      #

      resources:

        requests:

          storage: 40Gi

[Jim Knepley - ReleaseTEAM]

I would try using azuredisk instead of azurefile.

This thread isn't exactly on point, but apparently azurefile can be very slow in some situations.

Azure managed disks are block-level, which I strongly suspect is more appropriate to a Lucene index.

[Eloi Serret]

Hi Jim,
I tried with AzureDisk as Premium and it seems to improve a little but not enough. Now I'm trying UltraDisk to be able to use high IOPS and it seems to run better. I'll be testing different IOPS range to try to find the best config.

I'll be updating, but meanwhile thank you so so much :)

I must say that it's disappointing that there is no oficial documentation about the best way to implement Jira Kubernetes in an Azure environment.. specially when the Jira is XXL.

Best regards,
Eloi.

[Eloi Serret]

Hi Jim, just updating you, UltraDiskSSD seems to work better. I tried upgrading a 100Gi disk to 20.000 IOPS and the ScriptExecution time of copyProject has reduced to around 6 minutes..

It is still more than suposed to be, but it is so much better of 30-40 mins that was with AzureFiles.

I'll be investiganting further and report it to Atlassian Team, but for the moment I'll accept your Answer as Accepted.

Thank you so much :)

[Jim Knepley - ReleaseTEAM]

We've been focusing on the SAN side of that equation, but there are limits based on the VM size and type, too. 

Atlassian has a benchmark that we probably should have run to begin with, that might highlight performance shortcomings.