What is the Jira instance strategy at large companies?

[Eric Longoria]

Quick introduction: I am a Project Manager at a company with more than 3,000 employees around the globe.

At this company, we use one single instance of JIRA 5.2.11.  Ultimately, my question is: what is the JIRA instance strategy used by other large companies?  Do they also have only one single instance of Jira? Do they group similar projects onto a few different instances of Jira?  How do they determine who is allowed to have Global JIRA admin rights? How do they pay for plugins?

If you want to know why I ask this question, it is because using JIRA at my company has frustrated me for three reasons: poor performance, lack of control, and costly plugins.

Poor Performance

I think that having so many users, issues (1.8M), and projects on a single instance causes performance problems. Pinging servers in one location behind several security measures adds to this problem. Rapid boards, updating tickets, and running reports are all very slow.  JIRA has not been performing for us at all, and I want to know if there is something that we can do on our end to fix that, or just accept it as a shortcoming on Jira's part.

Lack of Control

Because our HR and Legal teams store sensitive information on our one Jira instance, IT limits the people with global Jira admin rights to just a very few.  That means that if we want to add one value to a custom field, we have to fill out a Jira ticket that generally goes unanswered for a day or two and then is resolved within 30 seconds.  I would like to have global JIRA admin rights so that I can change custom fields used only by my project, and make workflow changes when I need them instead of having to go through IT. 

Costly Plugins

Also, installing new plugins is both a pain and costly.  Because the instance is so large, the plugin is usually non-performant.  Very few companies test their plugin on a JIRA instance with 3000 active users and 1.8 million issues.  Plugins are also expensive because we have to purchase several thousand seats. 

I appreciate any and all help you can offer!

[Doug Swartz · Accepted solution]

The presentation Nic points to gives excellent advice. Especially don't make performance driven changes without measuring and knowing why you're making the change.

I work for a good sized company with a decently large, but not humongous, JIRA infrastructure. 3500+ employees worldwide. Five or six JIRA instances, including a 2000 user instance in the US, 500 user instances in Australia, and South Africa, etc. The US instance currently has 200,000+ issues, growing at approximately 15-20K/month and 200+ projects. We routinely have about 600 concurrent users during peak hour and 1500 http requests per minute. I'll lay out some of the things we've done and why. Maybe they will help you start to think about how to approach your challenges.

Plugin purchase

We use an approach where there is a small (6-8) group of people who must approve plugin purchases. They make sure the plugin meets various criteria including plugin support history, whether it duplicates the capabilities of an already purchased plugin, etc.  We've worked with Atlassian to co-terminate all of our plugins on the same date. Since you can't buy a plugin for less than 12 months, this means the initial outlay is for the first year plus 0-11 months. We make the requesting organization fund the initial purchase of the plugin, but subsequent renewals are funded out of a corporate support cost center.

Here's some info about different performance challenges we've had and what we did to address them.

Problem: People a long distance away from North America get much worse performance than North American employees

Solution: Measurements show bandwidth OK, but network latency was awful (sometimes 100 ms) between the US and both South Africa and Australia. In general JIRA is not high latency network friendly.This was especially bad when we connected to Crucible and far away repositories (don't even try this)  Network accelerators didn't help much because of all the dynamically built portions and AJAX-ness of JIRA screens.

We decided to have JIRA instances "network close" to the people who primarily use the instance. Additionally we implemented VDI desktops in the US for people from far away who need to access the US instance. JIRA access from a Citrix desktop has significantly better response time over a high latency network than accessing JIRA directly from your desktop browser.

Placing JIRA instances close to people on the network also means they are generally close in time zones. This is very useful when it comes to scheduled maintenance such as a JIRA upgrade. The outage to upgrade from version 6.0 to 6.2 is now in the middle of the night when few people are using it, rather than being in the middle of the day for a bunch of people. With a local JIRA administrator it also gives people everywhere good response to their needs.

Problem: JIRA responds very slowly during peak times. Dies during the afternoon every couple of weeks.

Solution: As might be expected there are multiple things that contribute. JIRA logs, garbage collection logs and the JavaMelody monitoring plugin are invaluable to deciphering what is going on behind the covers.

Java garbage collection logs show us doing minor garbage collects multiple times a minute and full GCs about every couple of minutes. Full GCs take 10's of seconds, stopping all user work.

A home-grown plug-in is found which occasionally used excessive (100's of megabytes) of heap space during a long running transaction. This is fixed

Max and initial heap size is increased from 2 to 6 gigabytes

The garbage collector is switched from the default to the parallelOldGC collector.

These changes make a huge difference. Full garbage collects now only happen about 2-3 times an hour and they only last 4-5 seconds. But about once a month, the system starts to degrade, with more and transactions hanging up in a database lock situation until JIRA dies or is killed to put it out of its misery.

Further research finds that when JIRA was initially installed, the Atlassian instructions for setting up the database connection for SQL Server were not followed. After these settings were corrected, the system has been running very well for several months.

Conclusion

1. Your mileage will vary.
2. Performance problems often don't have one simple solution. They should be approached using measurements and usually require more than one iteration.

[Holger Schimanski]

Concerning Lack of Control, we want to control the number of custom fields, workflows, issue types, because we see value in discussions about new ideas, problems and new approaches to learn as an organization. To organize this discussion we have a discussion forum and a JIRA project to track bugs, change and feature requests and also change advisory boards (CAB) with key users as delegates from different groups and roles in our organization using JIRA.

Thus, only a few people are JIRA admins for implementation of these changes and for operations like project creation. All other changes on workflows and custom fields go through the change advisory board. The strategy of the CAB is to limit the number different custom fields and workflows, but also implement enough flexibility to support different situations in the organization.

[Paul DeSousa]

Though I am a much smaller instance (500 users) I understand your concern.  My recommendations are:

Performance: I recommend you open a support ticket at support.atlassian.com they are great

Lack of Control:  I completely agree, I would like to have users be able to create custom workflows, screens, fields, and more.   I will create a feature request and it would help if you voted on it. 

Plugins: Yeah sorry no opinion on this one.  I usually try to use only free ones and am learning to write my own (though it is quite a steep curve so far)

[Nic Brough -Adaptavist-]

You'll probably find your issue closed as a duplicate of https://jira.atlassian.com/browse/JRA-3156 As it stands, a good recommendation IS to keep the jira admins to a very small group - three or four for most installations. If you think you need more than 10, you have a structural problem. On the other hand, I'd be quite happy to let big places have more admin rights - it'll keep me in a job for years - almost every Jira engagement I've had for existing installations has started with "well, your first job is going to be to clear up, it's a complete mess because we allowed lots of people admin access and they did what they wanted without talking to the rest of the business"

[Eric Longoria]

Awesome response, thanks, Nic! That takes care of the "Lack of Control" frustration, how about the "Performance" and "Plugins" questions?

[Nic Brough -Adaptavist-]

Mmm, I completely agree with your worries on those to be honest. I don't have a huge amount to add. Keep plugins to a minimum, look for the ones that you *really* need. As you say, many have not been tested at scale in any decent way (there's at least three in the marketplace I'd run screaming from for large Jiras, not to mention a couple I wrote that you really really shouldn't install) As for perfomance stuff, it's not quite my field. I know quite a few tricks that can help inside Jira (keeping things simple and consistent, disabling stuff that causes load, etc), but at scale, there are two things I recommend now. 1) Data Centre. You'll need Jira 6.2+ to do it, you are a bit more limited on the plugins you can use (the ones I'd happily recommend for almost all Jira installs are all on that list though), it's going to need planning, hardware and quite a lot of effort, but it could make a huge difference 2) Get an Atlassian partner to help you review your usage and tune it. In the interests of openness - I work for Adaptavist at the moment. We have some experts in making Atlassian stuff go more quickly, and even before I joined them, I'd happily recommend them to anyone with performance issues. Before you go looking for any experts, watch https://summit.atlassian.com/archives/2013/inside-the-massive-team/the-not-so-dark-art-if-atlassian-performance-tuning and see if you can learn anything from that (I sat through it six times, thanks to rehearsals). I'm not a performance expert. Dan is.

[Radek Antoniuk]

Nic and others have already replied very thoroughly. I will add only another possible possibility to use. You can use several instances of JIRA, depending on your location or anything. Then, even if it happens that you need the projects synchronised between them, you could use: https://marketplace.atlassian.com/plugins/com.atlassian.cpji.cpji-jira-plugin I totally agree with Nic's statement about lack of control. I am rather narrowing the group of people being able to create custom workflows or issue types, because the process in the company gets blurry and often people are re-inventing the wheel not the very optimal way.