Since we upgraded from Jira 7.4 to 8.1 we had major slowdowns for weeks. After this it stabilized but performance was still lower than 7.4. We had a planned power outage this weekend and I am affraid the issue is back.
- Healthchecks are fine
- Checked logs
- Disc IOPS are within range
- Network usage is well within order
- Everything under system -> troubleshooting an support tools checks out.
- CPU usage is 1% and 30% of 32gb ram assigned is available.
It takes several minutes just to login or open anything on the system, confluence and bitbucket is also affected. For example, background re-indexing normaly takes 5-10 minutes and now two hours or so.
I have logs in zipfile but not sure what to look for or how to troubleshoot this massive performance loss. Can see what the app is doing/where it gets stuck?, any help would be much appreciated.
We are running Atlassian on CentOS 7.
Maybe the below is the issue?:
30-Jun-2019 10:00:36.929 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/data/tmp/atlassian-jira-software-8.2.2-x64.bin.23598.dir/user::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
30-Jun-2019 10:00:37.081 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
30-Jun-2019 10:00:37.095 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 511 ms
30-Jun-2019 10:00:37.104 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
30-Jun-2019 10:00:37.104 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.40
30-Jun-2019 10:00:37.114 SEVERE [Catalina-startStop-1] org.apache.catalina.startup.HostConfig.beforeStart Unable to create directory for deployment: [/data/atlassian/jira/conf/Catalina/localhost]
2019-06-30 10:00:40,632 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/data/atlasian/application-data/jira' is validated and locked for exclusive use by this instance.
2019-06-30 10:00:40,823 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger]
Hi @Christian Radwanski ,
you can use the support-data.sh script from Atlassian to check your server : https://bitbucket.org/atlassianlabs/atlassian-support/src/master/
It will check disk speeds and get thread dumps, which you can then provide to Atlassian support to investigate further.
I recommend at least having a look at disk speeds : https://confluence.atlassian.com/kb/test-disk-access-speed-for-jira-server-performance-troubleshooting-818577561.html
Let me know if this helps,
Learn how to use two new reports for next-gen projects in Jira Cloud: Cumulative flow diagram and Sprint burndown chart. Ivan Teong, Product Manager, Jira Software, demos the Cumulative ...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events