What archiving strategies do you have for Jira? Share your tips!

[Shana]

Hi there, Community members! I'm Shana, a member of the Jira Software team.

Our team is interested in learning how you approach archiving in Jira. When issues or projects are done, do you remove them from Jira at any point? What about issues you might want to search for later?

We'd love to hear any strategies, best practices, or lessons learned! "We don't archive anything from Jira" is a perfectly acceptable response, too.

[Alexey Matveev]

Hello,

I like this KB about arhiving. Kindly read it:

https://confluence.atlassian.com/adminjiraserver071/archiving-a-project-802592917.html

[Shana]

Hi @Alexey Matveev, thanks for your reply! I'm familiar with this (very helpful) KB article, and I probably should have been clearer in my initial questions.

I'm curious not just how you might archive projects and issues in Jira, but the broader process around setting up an archiving plan or strategy. Who on the team needs to be involved? What questions do admins ask their stakeholders? What rules do teams set for themselves around when and why something will be archived from Jira?

I'm sure these strategies vary a lot from team to team, so I'm curious to see who has worked through setting up a plan like this and what they learned along the way.

[Bastian Stehmann]

We are discussing this topic often with our customers, because we did not find the one, best solution yet.

Our customers are often facing the problem, that they want to do some housekeeping to keep their Jira instance clear but they must keep records for several years.

In most cases we use one of the following ways.

If the project data must be kept, but it is unlikely that it will be searched again, it is exported via addon and then deleted. Problem with this way is, that the backup can be only imported in a jira instance with the same version where the backup has been made, so the installer for this must be backuped also.

The other way is to just remove Editing or even Browse Permission, and set a appropriate category, so that it is still in jira available an can either be always searched or be made easily available for searching.

[Shana]

Hi @Bastian Stehmann - thanks for your reply!

Both are very interesting approaches - that's good tip for everyone to know that a backup archive instance needs to be the same version as the production instance.

For the second strategy, it sounds like you would be updating the permission scheme, but leaving all of the issues and the project in the production Jira instance? Just finding ways to turn on/off the search for those issues and project.

I'm curious which add on you use to export projects from Jira?

[Bastian Stehmann]

Hi @Shana,

yes, thats right. When we use the second strategy, everthingy is left in the production instance, because we do not have found an quick and easy way to get the data back. We change the permissions to switch on and off visibilty of the projects, so it is just one click to make it possible to search some issues and one click to hide the project again afterwards.

We use the Project Configurator-Addon for exports of single projects.

[Jonathan Chatwin]

We use Botron Configuration Manager to take a snapshot of the project - configuration and issues and then delete the project and issues. If it ever needs restoring the versions are supposed to match in Jira but we find that it works even though the versions are different. It does however bring back custom fields and other config items so we do the restore to a dev instance first. 

For longer term projects where we need to keep them alive for audit reasons they just sit around in Jira. Would be nice (@Atlassian) to be able to archive these somehow and not count in the indexing and custom fields etc.

We've been investigating an archive server running Jira software (keeping it up to date)and just putting our Botron CMJ snapshots there...

Anyone have experience in this area? 

[Martin Gregory]

One problem with archive-by-hiding (permissions) is that it leaves all the old workflows and workflow schemes active.

This becomes a mess after a few year's worth of new versions of these...

[miikhy]

Hey Shana! 

Hope you're well!

I'd refer to the KB as mentioned by Alexey! That being said, to answer your broader question, I've recently faced it in critical environments and there were many people involved! Obviously Atlassian team must be part of the discussion but system admins are a great asset, sometimes legal or any related department might be involved for Intellectual property management and critical information backup strategy.

All concerns can be addressed either by hiding the project, backing it up or making it read only, as per the KB, the best track though might be found while getting all actors around the table and following regulatory requirements first!

The "best" case I can remember was a customer backing up the virtual machine (DB + App) everytime a project was archived, pretty straight forward to get access back to the data (restore is also quite easy), but that's definitely not the best approach in terms of disk space and environments management :)

Cheers

[Shana]

Thanks Micky! Doing well, and hope you are too!

Great tips - I like the advice to make sure legal is involved since there might be sensitive data stored in Jira that should be treated carefully.

[Mark Montminy]

I struggled with this at my prior company, and moderately so with my present (mostly due to Bugzilla).

For "near term" archiving, I do as has been mentioned here, mark the projects read-only, categorize them, and "park" them online.

Long term, that's a bit problematic as it retains ties to cruft you might want to clean up, or more importantly, doesn't make it obvious that these ties exist. Removal of a seemingly unused custom field for example could expunge data desired in those archived projects.

I'd love to have an export process that would cleanly bundle up a project with both a reasonably human readable form, say a tree of static HTML with proper relative links, as well as a computer process-able format, such as XML/JSON.

Then we could remove these archived projects from JIRA, while retaining the data in a usable fashion, freeing up the stale fields, addons, workflows etc so we can keep the system tidy and performing.

Granted, such an archive doesn't accommodate the potential desire to return the data to JIRA, at least not without some effort, but it's a start.

It feels like I've been asked this question from Atlassian for years now, and they're trying so hard to build the ultimate solution, that no solution is getting built :)

[Shana]

Hi @Mark Montminy, thanks for describing your current and desired future state for archiving. I agree that there's no perfect solution, but with input like yours, hopefully we'll make progress on addressing these requirements in the future.

[Deleted user]

I have been looking for best practice opinions on this topic for an ITSM implementation of JIRA SD. We don't have support projects that end, so there is no point in time we can really say this support information is no longer needed.

Questions we are asking ourselves at the moment trying to come up with a plan:

• How long back do we need to see specific issue details for a client? 
• We copy the data to a data warehouse nightly for external report tools, should we rely on this as a historical record and drop issues after x time in JIRA?
• Maybe we don't care about the details of an issue after 2 years and we should instead keep a set of monthly measurements in our data warehouse (counts of resolved by, impacted services, resolved by team, etc)
• Do we need a validation process to ensure things are committed to knowledge before we drop an issue? Do we assume after x amount of time this is just a given if it was needed (we have a knowledge management process that really answeres this for us I suppose)?
• How many issues sitting around in JIRA is too many, should we not care how large the number of issues sitting there is if they are resolved? (~450,000 issues after 1.5 years of Support use)?
• What's our goal in ticketing? Whats the goal for resolved tickets sticking around?
• If we decide we do want to remove issues, but not remove the project how would we even go about this when talking about such large numbers of issues?

I suppose the consideration and requirements for a Service Desk implementation are rather different than software development/project management. But still a topic I'm also trying to come up with a plan for.

[Étienne Borella]

Hi

On our side we always go with the online backup as described in the KB mentioned by Alexey. When we identify that a custom field, workflow or any scheme is no longer used by any "live" project we rename it with OLD_ at the begining to move the clutter to the bottom of the admin interface.

One issue that we didn't address yet is how to archive the attachments that stay on the server and eat up more and more disk space. I didn't have time to do a thorough research yet, so any tip about that would be very helpful :-)

Thanks!

[Martin Gregory]

How do you rename workflows?

[Gezim Shehu [Communardo]]

@Martin Gregory  maybe you mean you can't rename an active workflow.

Simply set it as inactive or create a copy and then rename.

[Martin Gregory]

But you can't set it inactive because it is used by the projects that are being archived...

One theoretical possibility is to make a copy with a better name, then assign that copy to all the projects.  But that would be a night mare of laborious work, given the number of projects that use old workflows...

[Brandon Selberg]

So from a business perspective we have requirements for a corporate retention policy. The stakeholders are auditors, devOps, business users and project management.
 
We have looked at the way described in https://confluence.atlassian.com/adminjiraserver071/archiving-a-project-802592917.html.
Most of these simply just hide the issue from the end users perspective yet provide nothing of a performance gain for a larger instance.

We tried exporting a project but even that has issue.
You have to tackle exporting all information and then we can’t use that project again. Then we ran into issues of viewing these later and searching them.
Consider this: What do you do when you have a long running project that has 30-60K issues a decade however you can’t retire the project?

Deleting projects entirely is also not a viable solution. When you delete a project all externalized links break to the referenced issue. This isn’t that big of an issue however making a new project sharing the same old key cause other issues.

So the real question from our team was, “How do we clean up issues from the main instance while maintain the same level of visibility?”

Answer: well it can’t be on the same server without impacting indexing performance and general clutter.

Then the follow up question is “How can we offer the same functionality in a read only mode of the main instance while preserving the original information while not being the main instance?”

Answer: Run another instance in a read only mode and find a way to get issues/content from A-B.

The A-B has been solved through some home brewed solution. I cannot give detail on how we accomplished it but we would love for a product native solution to be provided.
Preferably one that scales for those who have over 500k issues in the instance.

[Dom Baldin [Adobe]]

We need a read-only version of Jira. This would serve us two-fold. We could direct all reporting needs to this read-only Jira, offloading the load from our Production instance. Our performance is getting hammered by large API queries and eazyBI data imports.

A read-only Jira would also be able to serve as a nice Archive instance.

I've got all kinds of ideas! :)  

[Brandon Selberg]

read only is pretty easy. if your lazy it can be as simple as xml backup. import to another instance. run an API call stripping all permission groups aside from browse. boom read only instance. run script 1 x a week. 

[Dom Baldin [Adobe]]

We have a replica of our Prod DB that is replicated instantly, and we want to offload all of the traffic to the replica. Jira would need to be able to run on a DB that is read-only. Is that possible?

[Brandon Selberg]

We couldn't get the service to stay alive on a read only. We had to make all edit lockouts from an application layer. 

[Jason Michaud]

Hi Shana,

"We don't archive anything from Jira"

We are in a similar boat as Dominic Baldin. We use Jira mainly for the JSD project. We went live in early January and we converted about 200,000 existing tickets over. We are currently up to 220,000 issues logged for this project. I think we have a ways to go; but I'd also like to know of some statistics on when is a good time to think about archiving or moving some of the older issues out. This has to at some point put a drain on system resources and indexing. If we migrate issues to say a separate project outside of JSD - would that have any sizable impact on Jira performance.

[Gordon Andersen]

Hi Shana,

Does Atlassian and other in the thread have a GDPR take on this matter?

When archiving issues you need to make sure that the issues you backup are GDPR compliant which makes it (for some companies), hard just to backup a project without screening it.  

[Martin Giertl]

Hello,

in one of our solutions, we were forced to think about archiving because the instance started to produce 12x more tickets than originally planned and anticipated (this is actually a very good thing from the business perspective, because the issues reported now existed before the Jira/JSD implementation as well, but were not reported due to the terrible UX of the original solution - Good work Atlassian JSD team! ) and by this pace, the HW/OPEX costs would be unacceptable for the customer (we are talking some 1500 tickets a day in both JSD and Jira core - the solution combines those two). 

The solution of this situation was as follows:

1.) The analysis with the customer's business representative revealed the fact, that all the tickets older than XY months after the resolution were not interesting for them anymore (some mandatory data used for long-term reporting were already in the data warehouse and every analysis of the root cause or an entry about a known problem is already in the confluence, otherwise the ticket couldn't be closed).

2.) A three-stage archiving process was defined:

* Move the tickets resolved before 3+ months into the archive project (the workflow of the archived tickets is simplified etc.). An entry in the data warehouse describing the ticket is created in this stage. These tickets can still be restored. Only selected group of users can access the archives. Tickets are restored on-demand.

* After some time in the archive, the data is reduced (stripped down of anything that is not mission-critical, or not interesting for the reports). These tickets cannot be restored anymore (conditions prohibits this), but the issue history is still present.

* After another period of time, the ticket is removed altogether.

The technical realization:

All three parts of the process are implemented using jira services. The services are provided by a custom add-on. In the service configuration dialog we can set all the necessary parameters as the time differences, source and destination projects etc.

The services run overnight and are load-balanced (in some basic way, processing the tickets in small batches, providing small pauses to sync indexes etc.). Jira's internal planner is used to run the services.

As the example shows, the archiving approach is always heavily dependent on the customer's needs and strategy. Hope this helped/inspired you.

[Sreenivasaraju P]

We are also facing difficulties to find the appropriate method for archiving. 

We are using Offline archival method . Previously we used to identify (filter out ) the data which can be archived. At the time of archival we used to take snap shot (backup xml/db) of the production server and restore in archival server and update is as read only. so users can view the data from archival instance but can't update the data. From production server we used to deleted identified/filtered requests. So the number of issues used to come down. 

We have followed this process 3-4 times, now we facing difficulties maintaining these many archival instances. Our archival servers are in different versions of JIRA (4.x,5.x,6.x ..etc). 

We are archiving issues only. we are not archiving the project. Mostly we are arching the requests which closed a year before. 

Even though we are having multiple archival instance we have installed all these instances in one server with different ports. Our latest archival instance will be running. If any user requests for any archival data, we used to run the corresponding instance. Later we will stop the instance.  Monthly we will 5 requests for archival data (on average).

We are looking for some alternate/best approach for archival process.