Jira Slow at Specific Times

[Jh]

We are having an issue where Jira is very slow/unresponsive during a specific time-frame every day. During this slow-down time, creating, updating, and commenting tickets take a full minute. Loading between pages or projects is either very slow or throws "Internal Error Occurred" errors. This is a Windows 2016 VM on vCenter and it has 12vcpu and 32GB RAM on a Performance disk. We have increased the JVM RAM as well. When we do a disk access speed test, they are very high during non-peak hours.

We have ruled out automated backups, antivirus, and contention with other applications on the same VM. Anyone have ideas besides moving to Linux?

[Brant Schroeder]

@Jh 

I have seen this on windows machines caused by many different things.  I would like you to answer a couple of things for me.  

1. During these slowdowns what is happening with the resources?  If the CPU really tapped or the RAM?
2. Is your DB on a different server?  How many threads are open on Jira and on the DB?
3. Is the DB hosting other applications, DB, etc. that could also be taxing the connections/threads.
4. I know that you said you eliminated anti-virus.  Can you confirm that the anti-virus is not doing real-time scans.  I have seen anti-virus kick off when new files are uploaded and during peak times when the uploads are heavy this has caused heavy CPU usage on the server and threads to slow down making the system slow.

Look forward to getting additional information back.

[Jh]

@Brant Schroeder 

1. CPU gets tapped, not so much memory
2. DB is on the same server
3. Can you clarify what you mean by how many threads?
4. Not hosing other apps
5. I un-installed AV to test

[Brant Schroeder]

@Jh 

Great so we know that the CPU is the issue if it is tapped when you are having the slowness.  The CPU only has so many threads to process information on.  The CPU is serving both the server and DB which could be the issue, you might not be sized correctly.  Couple more questions.

• Is the DB only hosting Jira or do other application use it?
• Is the server only hosting Jira and the DB or are other application running on it?
• When the CPU spikes what applications are using all the CPU?

In regards to DB threads and DB threads in Jira it is how many connections you are allowing.  This can be an issue if there are not enough of them.  https://confluence.atlassian.com/adminjiraserver/tuning-database-connections-938846864.html  The information you provide above will help determine the issue.

[Jh]

@Brant Schroeder 

1. Sorry, I was mistaken, it seems both CPU and RAM are tapped. Task Manager shows both around ~60% load.
2. DB is only hosting Jira, no other applications use it
3. Server is only hosting JIra and the MSQL DB
4. The Apache Commons Daemon Service Runner with Atlassian Jira service is using bulk of the CPU.
5. The link you sent does not work unfortunately.

[Jh]

We have 21 connections set and R/W don't hit max very often.

[Brant Schroeder]

@Jh 

60% is actually acceptable on RAM.  Sounds like the processor is working a bit hard,  Is it at 60% all the time or does it spike up higher during slowness?  The DB actually looks healthy and is not taking a hit so that is not the issue.  Is the CPU dedicated on the VM or is it shared?  If it is shared then other VMs on the stake could be affecting your application.

[Jh]

@Brant Schroeder Normal CPU is very low, below 10%, RAM around 60%. CPU is shared on the VM,

[Jh]

What is your recommendation regarding the Shares/Reservation on the VM?

[Brant Schroeder]

I would recommend reserving the minimum that is required for your Jira instance size https://confluence.atlassian.com/jirakb/jira-server-sizing-guide-975033809.html

I have seen slow down on Jira VM instances when the shares are used by other VMs during load when reserves are not in place.  Give that a try and see if that fixes the issue.  You should be able to look at the VM logs and see if the slowness correlated to CPU usage by other VMs.

[Jh]

@Brant Schroeder, looks like we already have the VM specs recommended for our instance size(Med-Large, 2000 users, 50 projects.). During the time of slowness, it is in the middle of the night so maybe it's related to vMotion activity or something.