Who has experience with > 200k issues in Jira? Was performance degraded?

[Nancy Belser]

We have about 115k issues in our jira 4.1.1 instance. I am concerned about the stance Atlassian has made with regard to Jira performance at around 200k issues. Their recommended solution is to split the Jira instance at that point. For us, however, that would be a major undertaking because we have outside scripts and systems that run against our Jira instance as part of our build/release process. Some of these processes require linking issues that would likely end up split between instances. One of our scripts in particular needs to use the issue links table to follow the link hierarchy - that would be pretty difficult since Jira proper cannot even link across instances. What our your experiences with Jira performance with >200k issues?

Atlassian, do you have plans to increase Jira's scalability so that splitting the instance is not the only solution? There must be other customers with growing Jira instances who do not find splitting to be an acceptable answer. Isn't support.atlassian.com already at or near 200k?

[Wojciech Seliga · Accepted solution]

Hey,

Scaling JIRA to many hundred thousands issues is a complicated subject. Definitely this 200K is not a hard and strict limit. It's just given as a rough estimation of when you may hit some performance problems (so that you think about it in advance as you do).

Sure, there are companies which have more than that. Some of them experience various performance problems, other not. It depends on a lot of factors: number of fields, plugins used, users, concurrent requests, web server settings, OS and IO throughput, etc.

For one of customers who uses JIRA as a help-desk system we scaled JIRA to more than 1 million issues and did thorough tests in our lab. We wrote a blog post with some technical information about it. You may find it useful.

Cheers,
Wojtek

[Nancy Belser]

Thanks, I am reading your page now!

Nic, below, refers to the effect of having a lot of users. I noticed you had only 14 concurrent users for your test - how did you arrive at that number for the test? We have about 700 users, but I am not sure how many are actually active at any particular moment. We leave user sessions up for 24 hours and can have 1000 or so alive at a time, but that doesn't they are all doing something. Also, are you using the 64 bit JVM (assuming you would have to be). Any trouble making that switch?

[Wojciech Seliga]

Hey, 14 concurrent users here were used due to specific requirements of the customer (JIRA as helpdesk) - they have a lot of issues flowing through their JIRA daily (thousands) but only 10 - 20 of support engineers. 14 concurrent users was a little bit pessimistic approximation of the predicted load generated by these users (not every support engineers access JIRA all the time).

What we observed however, was when we increased the load to 50+ really concurrent requests - JIRA performance degraded seriously - mostly because more time-consuming requests (like full-text search or complicated reports) were dominating the pool of available threads serving web requests.

But please note - 50 really concurrent requests does not mean 50 active user accounts. It sometimes 200, sometimes 1000, sometimes 100 000 users - everything depends on how frequently on average these users access JIRA instance.

Looking at your HTTP access log should help you determine the ratio between all users in your system and the number of users concurrently accessing your JIRA.

[Nic Brough -Adaptavist-]

I've been there too. Given decent hardware, we racked it up to 500k without problems, in quite a complex install.

Your main problem is likely to be load induced by users - if you have a lot of them, and they're doing large reports you're going to get problems (or more to the point, inefficient ones - exoorting filters over and over into excel and reformatting because they can't be bothered to use Jira properly and are more comfortable with introducing errors into excel).

I've also found that the problems are mostly related to volume and complexity of data - lots of issues with lots of data in custom fields. It's not the fields in themselves that are a problem - having hundreds of them is fine, as long as there's not a lot of data in them overall. Plugins are a worry because some are not written to scale (I learnt this by writing one of those myself). Complexity of workflow, field configs, having a lot of issue types, complex mess of screens, notification schemes - no effect on performance, really, don't worry about that at all. One really easy win though - if you've got public stuff, then use "anonymous browse" rather than saying "user in group/role can browse project" - it means Jira stops needing to do any "can this person see these issues" work at all, which really can make it a whole lot easier.

[JamieA]

+1 for using the "Anyone" group rather than jira-users - http://blogs.onresolve.com/?p=95. Makes a huge difference.

[Nic Brough -Adaptavist-]

Sorry Jamie, yes, I'm 99% sure that's where I got the idea from in the first place!

[Nancy Belser]

Nic, what sort of hardware are you using? Did you go to 64 bit JVM? We are experiencing memory problems in our environment and I cannot increase memory any higher in 32 bit.

[Nic Brough -Adaptavist-]

Nancy, yes, although the large installs I've used were all on 64bit JVMs already, so I can't really tell you that that improves things.

[Jeremy Largman]

I've just written a document about this in our knowledge base: JIRA Performance Tuning. It's a simplified article for a complex subject, but it's based on what we've observed after lots of experiences of customers who are scaling JIRA. Ultimately, it is indeed our perspective that you should aim for splitting instances in the long run, but obviously that's not ideal, and that document is intended to help at least get JIRA tuned to some degree. It's not an exhaustive document; the blogs linked above are also really fantastic complementary docs. Hopefully it's a good start for most situations.

As for support.atlassian.com, it's not quite at 200,000 issues yet. We're headed there soon. We don't have lots of custom fields nor lots of issue links nor lots of bulk edit operations as a normal development oriented JIRA instance would (it's a support instance, of course), so it's not necessarily going to hit limits in those ways.

[rmk]

The published 200,000 issue limit was obtained a long time ago and things have changed a lot since then. We already know that the total number of issues is just one of many JIRA dimensions that can effect performance, some others include custom fields, active workflows, and simultaneous users. In the near future we plan to start testing different areas of JIRA so that we can give customers interested in large JIRA deployments more definite answers based on our performance testing to help you make decisions about scaling JIRA.

We are particularly interested in talking with customers who already have a JIRA instance with more than 200,000 issues. If you fall into this group please send an email to the JIRA Product Manager with the subject line: JIRA in the Enterprise and we will respond accordingly.

UPDATE - 25 June 2012: From JIRA 5.1 onwards we are dropping our previous guide line of splitting your instance once you are around 200,000 issues! We've always known that this is not a hard limit and now we have a great resource that will help you scale JIRA.

[David Corley]

Hey Nancy,

We're running a heavily used Jira instance containing 350,000+ issues with between 11-12,000 issues added/month.

We experience occasional short-lived GC pauses, but overall, I'd say performance was just fine.

I've posted as much information on what we've tuned here:

https://answers.atlassian.com/questions/22592/is-your-jira-instance-growing-above-200-000-issues?page=1#22614

Hope it helps.

FWIW, we have added a 2nd Jira instance, and are planning a 3rd. The 2nd Jira instance was not put in place for scaling reasons (but it will help), and the third instance is dedicated to automatically generated Jiras (by various business processes). We want to sandbox any machine generated Jiras in the event one of the generators goes crazy and floods Jira with creation requests. In this sense, the 3rd Jira instance is simply a sandbox.