Major slowdown after reboot

[Christian Radwanski]

Hi guys,

Since we upgraded from Jira 7.4 to 8.1 we had major slowdowns for weeks. After this it stabilized but performance was still lower than 7.4. We had a planned power outage this weekend and I am affraid the issue is back.

- Healthchecks are fine
- Checked logs
- Disc IOPS are within range
- Network usage is well within order
- Everything under system -> troubleshooting an support tools checks out.
- CPU usage is 1% and 30% of 32gb ram assigned is available.

It takes several minutes just to login or open anything on the system, confluence and bitbucket is also affected. For example, background re-indexing normaly takes 5-10 minutes and now two hours or so.

I have logs in zipfile but not sure what to look for or how to troubleshoot this massive performance loss. Can see what the app is doing/where it gets stuck?, any help would be much appreciated.

We are running Atlassian on CentOS 7.

Maybe the below is the issue?:

30-Jun-2019 10:00:36.929 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/data/tmp/atlassian-jira-software-8.2.2-x64.bin.23598.dir/user::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
30-Jun-2019 10:00:37.081 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
30-Jun-2019 10:00:37.095 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 511 ms
30-Jun-2019 10:00:37.104 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
30-Jun-2019 10:00:37.104 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.40
30-Jun-2019 10:00:37.114 SEVERE [Catalina-startStop-1] org.apache.catalina.startup.HostConfig.beforeStart Unable to create directory for deployment: [/data/atlassian/jira/conf/Catalina/localhost]
2019-06-30 10:00:40,632 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/data/atlasian/application-data/jira' is validated and locked for exclusive use by this instance.
2019-06-30 10:00:40,823 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]

****************
JIRA starting...
****************

[Alexis Robert · Accepted solution]

Hi @Christian Radwanski , 

you can use the support-data.sh script from Atlassian to check your server : https://bitbucket.org/atlassianlabs/atlassian-support/src/master/

It will check disk speeds and get thread dumps, which you can then provide to Atlassian support to investigate further.

I recommend at least having a look at disk speeds : https://confluence.atlassian.com/kb/test-disk-access-speed-for-jira-server-performance-troubleshooting-818577561.html

Let me know if this helps, 

--Alexis