TL;DR - GSAC has gained significant performance improvements on Jira 8.5!
Indexing is down from roughly 5 hours to 1 hour.
Transaction time is under 200ms during regular load.
CPU Utilization is much lower with far fewer rolling restarts necessitated.
Customer Support Technology and IT Operations have both put in a lot of work to get GSAC (getsupport.atlassian.com) onto Jira 8.5, and wanted to point out some of the performance improvements we’ve seen over the last 3 weeks.
Indexing improvements were a big deal in Jira 8 and we were very keen to see our own improvements. Historically our reindexing time has been in the range of 5-5.5 hours (even on a designated instance in the pool), which can be quite the problem when there are changes that need to be made which require downtime. The improvements have been overwhelmingly positive in regards to this as reindexing has taken a little over 1 hour while keeping itself much smaller than previous packaged indexes (saving us disk space!).
We have also had issues with index inconsistencies due instances attempting to copy over indexes from others. Although it is a little early to tell, we are optimistic that issues related to these corruptions will be far and few between!
There was also a good amount of time spent on improving performance for browsing, searching, and loading boards in Jira 8. I can’t remember the number of times that we have had reports of “GSAC is slow”, and we even had data to back those claims up. Well now it’s safe to say that although it may still be slower than some people like, it’s definitely a hell of a lot faster!
And these are the same transaction times measured from outside by New Relic.
The IT Operations team has been working tirelessly over the last several months or longer to determine what might be causing considerable CPU utilization on GSAC, resulting in countless rolling restarts in order to get whichever affected instance back into working order. CST has worked together with them updating plugins and gathering data to no real avail unfortunately… That is until we finally got to the point in which we could upgrade the system!
CPU utilization dropped dramatically to the point that we are planning to declare the issue resolved through the upgrade. Both teams had investigated every avenue of improving the problem, but we could never truly determine the cause (only that it seemed to be tied to the performance of Jira 7).
There was a lot of work needed from both CST and IT Operations in order to get GSAC in a place we could finally upgrade successfully, and we hope it lays solid groundwork for moving forward with a more consistent upgrade cadence. This included a few things on both sides:
Removing Atlassian Connect apps from GSAC
Investigating and removing incompatible apps from GSAC
Standing up and testing new stacks on Jira 8 for verification of internal apps
Upgrade or rearchitect internal apps for compatibility
This led to a lot of good work that can hopefully be used and laid out in a more streamlined manner for future upgrades.
I would like to give a heartfelt thanks to everyone that was involved in this massive endeavor including all of Customer Support Technology, IT Operations, and everyone in support and advocate roles who helped us conduct smoke testing and UAT before and after going live!