Jira (Spring?) cleaning

[#1]

As @Bridget mentioned - it is spring in Australia, time for house cleaning. Well, we, Europeans are half a year behind or forward (depends on perspective), but it does not matter. Rules of the game are the same.

So, let's talk on Jira cleaning

First things first - why would you do the cleaning? There might be several factors for, but two are the most important:

• governance - within it I would include security, unification, change control, program/project control, users and admins review etc.
• performance - within it I would include availability and capacity, utility of tools and so on

Your focus on what is more important might fluctuate, depending on actual, current needs and goals. Still, both need to be taken into consideration. Why? Proper governance cleaning will allow You to keep/improve performance measures, while proper performance based cleaning might affect governance by changing known, approved ways of working.

I did my time for one of the project running huge performance analysis for Jira DC instance. Instance performance was terrible, governance even worse. And first advise given was to clean it. Get rid of 75-80% of 1200+ workflows, 3000+ custom Fields, and a lot of other stuff. But to clean it wisely - was another challenge. We have spent 9 weeks on analysing thoroughly all aspects - from underlying infrastructure through architecture, technical and business configuration, ending up at usage patterns. For the purpose of this article I will leave behind Architecture and technical part, and focus on business config cleaning and usage patterns.

Business config cleaning

As You might know from Atlassian documentation there are multiple factors You need to maintain clean:

• Workflows & Workflow schemes
• Custom Fields, contexts, configuration schemes
• Screens and screen schemes (including Issue Types screen schemes)
• Permission schemes
• Notification Schemes
• Security Schemes
• Priorities, Resolutions, Statuses etc.
• Shared objects - JQL filters, filter subscriptions, dashboards, gadgets

What does it mean "keep it clean"? By my means - keep it as functional as You need, but structured, unified as possible, re-usable. Jira gives You great capability, as the open platform, to make every project uniquely configured. But gives you also even more important option - to reutilise and unify ways of working.

If I recall example above - instance with 1200+ workflows - 800+ of them were THE SAME workflow. used in different projects. Exactly the same, out of the box workflow. By using shared configuration You would downgrade volumetric of active workflows by 2/3s. Governance wise, and performance wise. 

Keeping the same example - avoid and correct need from admins, project admins, change requestors to leave their unique "legacy". In 3000+ custom fields in mentioned instance around 120 were different fields for "results description": analysis results, test results, QA results, UAT results, etc. And I don't mention split between expected and actual results. I mean lack of unified naming related to whole business configuration (there were different issue types for each of these activities, different story - also extensive too much). 

I will get back here to my idea that with systems like Jira - application administrator role is not enough. Either You, as admin, are capable of solution architecture design, or You need solution architect to discuss with key users and make decisions on proper use of business features. Admin will simply follow the request to add workflow, project, Custom Field, architect will question, analyse, justify everything accordingly to governance and to achieve performance.

There is a lot more of course. Assuming that documentation for mentioned project took almost 300 pages, this article is to short to go through every case. But to give You some more insight about important, tricky, or unexpected findings:

• 1800+ projects and 1800+ workflow/Field/permission schemes (little less - 70% of that - notifications schemes and screen schemes, and around 20% of that volume - security schemes) - almost every project had unique set - although after analysing - again - 70%+ had exactly the same configuration, but multiplied. 
• Longest workflow - 51 statuses/steps, with 47 allowing transition from all others, in total 10 WF with 40 + (!!!) statuses
• In total - around 300 text multiline fields - about 50% could be cut off based on unified naming conventions, another 30% could be switched to any select/checkbox/radio (and here is the tip - text fields have biggest impact on performance, as they are not indexed, but use full text search, with single ones - does not matter, but at large scale...)
• Natural, organic growth of Custom Fields volume per month at level of 20-30 new fields monthly due to misuse - set of fields was created for each biweekly sprint in one project (field names were referring to Sprint name, so every sprint had the same set of fields/data, but differently named) - that gives new 240-360 CF per year.
• 100+ Jira administrators with full rights to create new objects.

Bottomline - It was a mess to maintain, pain in the back for performance, and definitely not the instance You can use any governance on. By the end of the project - around 70% of findings were transferred into cleaning actions, limiting business config to around:

• 100 Issue types
• 150 active workflows
• 30 Workflow schemes
• 500 Custom Fields
• 20 unified permission schemes based on roles
• 10 notification schemes
• limitation of rights to shared objects creation and management
• and so on

Let's face it - I was not totally happy - I would cut it off more (especially WFs and CFs). But then it is still balancing between governance and performance on the one side, and utility on the other. 

Usage Patterns - important but often forgotten

What is often forgotten - is usage patterns. Partly it relates to previous paragraph - way of working within a teams and departments, and use of Jira features (especially shared objects).

There will be a number of things to consider:

• natural and occasional growth of projects/issues
• natural and occasional growth of users volumes

In all cases natural - related to extension of Jira into new business areas, occasional e.g. due to merges, acquisitions etc.

• Business and technical roles mappings
• Related to above user profiles (what is used, how, when, for what reason)

And here again - You need to avoid situations where You loose control over it (100+ admins - remember?). Growths can be anticipated, analysed, and adopted, if only there is a good communication and understanding of Jira's role in business processes (critical or non critical system, and core, enabling, enhancing role - I would say in most of cases it is not critical, but enabling - means - secondary just to critical systems). That means that governance over Jira health and performance is not one-off, neither occasional task, but ongoing process, requiring owner, rules, and enough effort.

Then - usage pattern themselves - they are transferable into roles, schemes, specific rights and so on. It has to follow identity management policies - and even in agile, distributed organisations - need to follow some ground rules, gathered in governance practice. You should keep standards, and base on standards for project roles and technical roles. Exceptions - as per name - are one-off setups for very specific, business wise needs. Let's make not mistake here - none of business mapping is so unique that couldn't be just variable from standardised way of working. If someone states otherwise - usually it is due to some tendency to run micromanagement, or shadowing the responsibilities. 

Micromanagement trends usually will result in will to keep projects totally under control of one-two persons, giving them all, including Jira admin rights, to do anything they want, regardless of sense. 

"Collective responsibility" trend on the other hand - drives to giving exactly the same rights (often, including Jira admin).

Both results in lack control over changes to specific projects. And then Your Jira holds 10 of them - it is not the issue. However, when it holds 1000... It is.

3 top tips for successful cleaning of Jira

Ending this story I would like to share with you 3 top tips I assume are the most important. And even if You consider them truisms, memorise them, and recall with every Jira cleaning operation:

1. Know Your instance - on daily basis, understand patterns, changes, know the reasons and habits of your users and admins
2. Re-utilise as much as possible inline with business needs
3. Keep ownership of configuration in strong, knowledgeable hands - with tech, and business architecture acumen.

Then You should be rather safe than worry.

[Bridget]

@PJ Wysota this Article is AMAZING! 

[PJ Wysota]

Oh, thanks!

[Kat Warner]

Great article @PJ Wysota . I feel much better about the metaphoric dust bunnies in the Jira system I use now.

[Shashi]

@PJ Wysota ,

Did you try to automate to clean up any stuff or was it manually done?

Any links to the article which you visited for clean up would be much appreciated

[Shashi]

How did you measure the performance? How did you collect those metrics?

[PJ Wysota]

Hi @Shashi

That is in fact a lot of questions in one-two questions.

1. Performance references from Atlassian - a number of them - main I used in my work:

https://developer.atlassian.com/platform/marketplace/dc-apps-performance-and-scale-testing/

https://confluence.atlassian.com/adminjiraserver/performance-and-scale-testing-965568707.html and similar for previous versions

https://confluence.atlassian.com/enterprise/jira-data-center-sample-deployment-and-monitoring-strategy-953148860.html

https://confluence.atlassian.com/adminjiraserver/performance-and-scaling-965568705.html

https://confluence.atlassian.com/enterprise/managing-custom-fields-in-jira-effectively-945523781.html

https://community.atlassian.com/t5/Jira-questions/Custom-fields-and-Jira-Performance/qaq-p/81234

https://confluence.atlassian.com/jirakb/troubleshoot-performance-issues-in-jira-server-336169888.html

https://confluence.atlassian.com/jirakb/common-causes-for-jira-server-crashes-and-performance-issues-203394749.html

https://atlasauthority.com/blog/how-indexing-postgres-and-flushing-caches-impacts-jira-performance/

2. As for performance tests:

- start with JMelody or alternative

- use tools like: https://community.developer.atlassian.com/t/jira-performance-tests-jpt-beta-is-now-available/23708

- use references like: 

https://confluence.atlassian.com/enterprise/atlassian-performance-testing-examples-946041214.html

https://www2.valiantys.com/en/jira-performance-at-scale

https://confluence.spartez.com/display/SPARTEZ/Testing+a+Million+Issue+JIRA+-+The+Verdict

3. Last but not least is maintaining performance/availability/capacity policies. And this governance practice is the toughest part, as You need to apply it from committed application management level. In simplest words it would cover:

A. Setting up plan/baseline (how, by whom Jira should be used as per designed implementation) + capacity/performance measures for this scenario 

B. If differs, actual/current use scenarios (how, by whom) + capacity/performance measures + gap analysis against planned/baseline

C. Setting up monitoring on tech resources to measure fluctuations and changes on server/application/database levels

D. Maintaining any changes to Jira use patterns (organic growth of projects/issues/shared components, additional projects/business units, volumetric additions of served users/customers, resource consuming add-ons or volumetric config changes) under restrictive change management process including performance testing for potential scenarios 

E. Maintaining system/app components - as usually newer versions provide performance improvements

F. Upgrading system, parametrisation of JVM, scaling up Jira or migrating to DC

[NotJoel]

Awesome writeup, keep em coming!

[Jess Seitz]

Fantastic article @PJ WysotaWow 51 steps in one workflow made my jaw drop, really interesting to hear how you weigh balancing governance with utility. What performance improvements were you able to see from this deep cleaning? I'm also curious if you were able to utilize project and issue archiving to clean up outdated issues? 

[civa7649]

I realize I am commenting nearly a year after this article was written but its content is still true and helpful.  I am in the midst of my own spring (fall?) cleaning and am finding similar trends.  From what I can see, we have many many workflows especially, that appear to be the SAME default workflow but with the new project's key in front.    Is this a default of JIRA?  Why does it make a copy of the default workflow with a name change?  

So I am tempted to go repoint the workflows to the original default and then delete the 'copy.'  But how do I know for SURE that it hasn't been changed.  Is it really a manual eyeballing of the workflow to confirm ?   Thanks in advance to anyone who can advise.  

[PJ Wysota]

Well, eyeballing would be the only sure thing. Or Insight for Jira with Jira environment add-on. 

But then, IMHO - I would repoint to original, make standard and announce, that it is a way untill someone has good justification to make amendments.

[Jousef]

Hey PJ, awesome article, we are using it frequently!

Do you have any tips for cleaning up Confluence as well?

We have almost 200 spaces, probably a quarter of them aren't being used anymore or redundant in some way. What would be a good approach to find out which spaces should be archived?

Would love to hear your or anyone's input!

Thanks and all the best from cloudy Berlin,
Jousef

[Gonchik Tsymzhitov]

Hi! 

Thanks for use cases I am going to enroll it here

https://github.com/gonchik/cleanup-scripts