How to export all issues from Jira in JSON format

[#1]

This article shows you a way to export all your Jira issues into Jira's JSON format and basically builds the JSON file format for you. If you have been administering Jira for a while, you would have discovered that JSON import using the external system importer is one of the ways in which you can import a complete project with the history. The below method uses a Python library called jiraone which can be installed using pip install jiraone or python3 -m pip install jiraone via a terminal.

See https://jiraone.readthedocs.io for more information about the usage.

In a previous article, I’ve shown how you can export all Jira issues into a single CSV file format, making it possible to export all project issues and import them in a single upload rather than multiple uploads of a 1K issue per file. This use case uses the same Python method issue_export() but expands on the arguments that can be employed in exporting issues into a JSON format. If you have been engaged in multiple migration projects as a Jira administrator you might have been wondering if it is possible to perform a migration from one instance to another in JSON format and how can this JSON file can be generated but on the other hand, creating this JSON file manually is a fool’s errand and it's prone to many trial and errors from a user’s perspective. For the longest time, I believe JSON import has been one of the most complicated processes to tackle when migrating to Jira whether it’s Cloud, Server or Data Center.

The process just seems too complicated, so a lot of administrators just avoid it altogether. On top of that, JSON import is an import that you can use to successfully import issue history, which is very vital if you want to retain all your data across multiple instances of Jira. While other methods of migration such as the Cloud-to-Cloud migration tool or the Jira Cloud Migration Assistance in the context of Jira Server or DC exists, this method allows the flexibility for you to download a copy of all your issues in Jira including the history. The tricky part of the entire import requires that no changes can be done if the item has been imported before which means only new items can be imported. So, once you get the JSON file make sure that the import that you’re undertaking works out in the first trial for the project(s) you hope to import into. This happens by ensuring proper project configuration is done on each project you want to import. Also, it involves making sure that you have a valid JSON output file that Jira will accept and validate with completeness. While I’ve always been keen on providing a solution to this problem, this has been on my to-do list for a while (years even) but then again I was reminded of it and I said why not, let’s get to work on it and lots of hours later, here we are with a script that does such.

The purpose of this API solves multiple problems with Jira migrations (exports and imports) such as a complete import from one instance to another including the historical content, a way to backup your data in your preferred storage service etc. Also, this method hosts the largest set of arguments in the jiraone library, as it has up to 30 arguments that can be used to tweak the export of issues either in CSV or JSON.

Let’s start off with the way to authenticate into the instance where you want to extract your Jira issues. You need to create a configuration file in JSON following the below example and save it as a config.json file. This will be our source of authentication.

{ 
"user": "prince@example.com", 
"password": "<APITOKEN_HERE>", 
"url": "https://nexusfive.atlassian.net" 
} 

Then you can call this configuration using the below Python script

from jiraone import LOGIN, issue_export 
import json  

file = "config.json" 
config = json.load(open(file)) 
LOGIN(**config) 
jql = "project in (AB, BC, IT) order by created DESC" 
issue_export(jql=jql, extension="json")

The above is a minimalistic way to export projects into a JSON format, notice the argument extension as this is vital when deciding which export you need. This method issue_export comes with a myriad of arguments which you can use to successfully tweak how Jira issues can be exported both in JSON or CSV format. However, in this article, we are primarily focusing on the JSON export part, which we’ll discuss further and how you can use this method with other arguments.

# If you want to export certain JSON properties in the export file 
# previous expression 
properties = ["users", "history"] 
# Then, you can do 
issue_export(jql=jql, extension="json", json_properties=properties)

The above allows you to include users and history export by defining the json_properties argument.

# If your export requires specific datetime format on datetime fields,  
# previous expression 
dates = "%d/%m/%Y %I:%M %p" 
# The above would translate into dd/MM/YYYY 09:14 AM 
# Then, you can do 
issue_export(jql=jql, extension="json", date_format=dates)

When importing issues into Jira, the JSON format in the DateTime field type expects the datetime in a specific format that allows Jira to understand the user’s time including the timezone.

# Alternatively, you can import date formats from jiraone 
# previous expression 

from jiraone.utils import DateFormat as df 

dates = df.dd_MM_YYYY_hh_MM_AM_PM 
# The above would translate into dd/MM/YYYY 09:14 AM 
issue_export(jql=jql, extension="json", date_format=dates)

If you do not know the Python transformation of the DateTime string directive, you can import some common ones from the utils module of jiraone

# Excluding certain custom field from the JSON file,  
# previous expression 
custom_type = ["com.atlassian.plugins.atlassian-connect-plugin"] 
# Then, you can do 
issue_export(jql=jql, extension="json", json_custom_type=custom_type)

The above should help exclude certain custom fields from the export especially those addon fields which might not exist in another instance. By default, it is set to remove cloud addons, so if you want to keep all custom field types including cloud addon types (if they exist on the destination instance), you can place the custom type in an empty list as below prior to triggering the script.

custom_type = []

If you’re querying a Jira Datacenter or Jira Server, use the below

# previous statement 
# change the API attribute to False when 
# working with Jira DC or Server prior to LOGIN instantiation. 
LOGIN.api = False 
issue_export(jql=jql, extension="json")

Although I did not test the Jira Server or Datacenter side, the JSON export should work (as the APIs are similar to Cloud in some aspects) to enable you to get a valid Jira JSON import file, which you can later tweak according to your needs by opening the file in a text editor of your choice and manipulating the contents. Let’s begin with all the arguments mentioned above and some that help to get better results.

The extension argument is the first argument that allows you to generate a JSON export and it is required for every other argument that supports the exportation of a JSON file format. It accepts two values either “JSON” or “CSV” which are case insensitive. In this case, since we require a JSON export, the value has to be “json”. Now that’s out of the way, please note that if you want to learn more about all the arguments in this API method or what this particular method does, refer to the documentation link mentioned above as that will give you a thorough explanation with some examples of what each argument does. Without much ado, let’s dive into the ones mentioned above.

Starting off with json_properties → this argument gives you the ability to specify if some objects should be included in the JSON export such as history items. The expected values are users, links and history and as a default, the history isn’t included in the export, you have to be explicit when it is required, the same goes for users and links. You can specify which one you want in the export by adding each value to a list. This argument values are case insensitive so you can send options as Users, LiNKs etc and it will work.

The date_format argument controls the way DateTime is rendered on the JSON export. It targets specifically DateTime custom or system field and re-renders the format so it includes a timezone which is the accepted format required for JSON import. If in a scenario, the date_format is not the default according to your own instance of Jira, the program will attempt to guess the format with a given list of other formats. If it cannot guess the format, it will raise an error and terminate the current session. You can also import some common date formats from the jiraone.utils module and attempt using such string directive as the date format.

The json_custom_type argument allows you to exclude a custom field from the export based on the field type name. This can be an exact string or the beginning of the string if it is unique. If it matches, it excludes the desired field. Subsequently, each issue will only produce custom fields that have a value, empty custom fields are automatically omitted from each issue within the JSON file to prevent errors or warnings during import.

There are also other helpful arguments that can make the JSON conversion easier. Below are helper arguments that perform some robust functions with the JSON export

• use_cache → This argument allows you to save certain objects and use them as the cache object for faster iteration after the first time the script runs if you are running the same JQL query rather than making the same HTTP call to your Jira instance. It expects a “True” value to be activated

• expires → This complements the use_cache argument as it extends the time the cache can be used. It expects an integer value to denote the number of seconds in the future that the cache should live for. e.g. 3600 = 1 hour which means after 1 hour if use_cache is true make a normal HTTP request rather than use the object saved as cache.

• sub_tasks → This argument helps you identify the sub-task issue type in your export and performs the parent → child relationship linking. Since the sub-task type can be created by a user, you need to supply all the different sub-task types by their name that is present in the projects you’re exporting e.g. “Sub-task bug” in a list to this argument.

  • sub_task_type = ["Sub-task", "Sub-task bug"]

  • If the argument isn’t supplied, it will default to Jira’s generic sub-task name which will be the only sub-task that it’s relationship will be added to the linking. If an empty list is provided, then no sub-task link will be created and any sub-task issue will become orphaned when imported.

• workflows → This argument helps you add the workflow scheme name to each project that is generated in the JSON file. Please add a valid workflow scheme name as an exact string. If the argument is empty, then the key to identifying a workflow isn’t added to the export JSON file. It expects a dictionary of project keys mapped to a specific workflow scheme name. When a project key matches, it uses such workflow scheme name in the JSON export
  • workflow = {"IT": "IT Simplified Software Workflow Scheme"}

• flush → The flush argument does only one thing which is to delay the time it takes for different running threads to shut down when extracting history items. It requires a number which indicates the number of seconds to delay. By default, it is set to 3 seconds. This allows for a complete history item to be added to the JSON export. Although the extraction will happen for all historical content, there might be 1 or 2 of the history content which might still be running or pending in a thread prior to the JSON conversion finishing. To prevent this history content not being added, this argument causes a slight delay which enables all threads to completely shut down if any are running.

With the above, you can comfortably export your Jira issues into a valid Jira JSON format and then import this JSON file into another instance with all data including history. This will give you the ability to start using the JSON importer more frequently than ever without worrying about how you would construct the JSON file. The logic behind this program handles most of the problems you can encounter and attempts to even guess certain formats on behalf; using simple algorithms. These are the little things to keep in mind while using the JSON import for Jira.

• You need to create manually your project schemes, workflows, statuses etc if it's not a default type of project configuration from the instance where the data is extracted. You can specify a workflow scheme in the JSON file export as part of the configuration.

• If you have any user-installed apps on your instance, make sure your destination site has the same user-installed apps too before you run the JSON import.

• Ensure that all issue types declared in the JSON file exist on the projects where you’re importing. If this does not exist, an invalid issue type might be showing up after a successful import. Albeit the importer will attempt to create it but there might be varying results

• The users mentioned on the JSON file must exist on the instance where the import is taking place else data won’t be imported properly

• It would be better if the custom fields already exist with their values in the destination site. However, the importer can recreate the custom field but not the options contained. Also, ensure that all needed fields are available on the screens of the project.

• Enable the security scheme or level in the project if they exist or simply add them afterwards if you’re looking for issues that were imported but don’t show up. This also can potentially break any parent → child relationship linking in sub-task

• Sprint values and boards must exist prior to the import.

• The history author doesn’t work with accountId, there is a MIG-376 which explains this problem. However, I remember back in 2019 prior to when Atlassian enforced accountId, that a Jira user could easily change their username or user key at will after that Jira users couldn’t change this aspect and Jira would simply pick up the user’s prefix of their email address as a user key. Therefore, you could potentially use the prefix of your user’s email address as the user key in the author section for the history item to get the right user visible with the JSON import.

If the above conditions are met properly, a high chance of you getting and importing your project into your Jira instance or another instance will be possible without or with minimal errors. Hope this opens channels for those administrators and users seeking to migrate their projects.

[Delfino Rosales]

Really interesting and useful, thanks!