How to export all issues from Jira via API

The below article will show you how to export all issues within your Jira instance to a single CSV file without any issue export limit. To use the below method, you need to generate an API token for your cloud environment and create a script using this python library called jiraone. You can install jiraone by using pip install jiraone or python3 -m pip install jiraone using a terminal.

advanced_search.png

notes.png

See https://jiraone.readthedocs.io

As a Jira administrator, you might have been plagued with the request of transferring projects across multiple instances owned by your organization or probably due to a re-org you’re tasked with managing the shifting of multiple projects into a different instance to differentiate the various subsidiary within your organization. This task however daunting must be solved by the administrator and managed for effective project deployments. With the below API, you can simply perform this task, not only does it save you time, you can have all your project in a single file for storage in a CSV format. If you think about it, this is a great solution for creating a complete project backup of your Jira instance on a regular basis.

The purpose of this API solves multiple problems. Just to name a few, the first is uploading a single document to the External system importer rather than multiple documents due to the 1K limit. The second is that your external stakeholders can now have a single source of truth for analysing or auditing your Jira projects in other spreadsheet programs such as excel or numbers. The third is that you can have a backup of your entire project stored in your preferred storage service.

Now constructing a program that reads accurately each row within a CSV file wasn’t an easy task. Keeping track of each cell, when they change due to the swap of headers and sorting each column accordingly from both files was a challenge which I embarked on. That’s why you see before you a method that allows you to download all your projects into a single file using JQL without any issue export limitations. That’s right, you can export even 10K, 20K or more issues if you want.

To begin, you will need to create a configuration file in JSON following the below example and save it as a config.json file. This will be our source of authentication.

{
"user": "prince@example.com",
"password": "<APITOKEN_HERE>",
"url": "https://nexusfive.atlassian.net"
}

Then you can call this configuration using the below python script

from jiraone import LOGIN, issue_export
import json

file = "config.json"
config = json.load(open(file))
LOGIN(**config)

jql = "project in (AB, BC, IT, IP) order by created DESC"
issue_export(jql=jql)

The above is a minimal representation to trigger an export using this reporting API into a single CSV file. If the export is done on a single file, you can now simply import the issues into another Jira environment in one single shot rather than in multiple files of 1K issue export.

# If you want to rename the export filename, 
# previous expression
export_file = "example_export.csv"
# Then, you can do
issue_export(jql=jql, final_file=export_file)

If you want to query each page and export only certain aspects of those pages, you need to know first the range of your Jira issues. i.e How many paginations are given to the total list of exported issues you want to export based on the JQL?

# previous expression
issue = PROJECT.issue_count(jql)
# result is a dictionary of issue count
# and max pagination of records pulled
# 1 page is equivalent to 1K records
count_issue = issue.count
# 18239
page_count = issue.max_page
# result
# 18

The above should give you an idea of what the max_page range is and how you can use it within the next example below.

# previous expression
issue_export(jql=jql, final_file=export_file,
page=(11, 18))

If you’re querying a Jira Datacenter or Jira Server, use the below

LOGIN.api = False
# previous statement
# change the API attribute to False when
# working with Jira DC or server prior to
# LOGIN instantiation.
issue_export(jql=jql, final_file=export_file,
page=(3, 6))

Now, let me explain the arguments that you can use within this method. These are the accepted keyword arguments that are used within the issue_export method. The folder argument expects a dir path in strings to a directory and allows you to specify which folder the dataset will be stored in. Please note that you do not necessarily need to have it set as it comes with a default directory. Usually, this should be a direct directory where you’re running the script. The jql argument requires a string following a valid JQL parameter. The final_file argument just basically allows you to name the final export filename. You do not necessarily need to have it set. 

When you use the page argument, you can selectively pinpoint which export range you want. The page expects a tuple of integers and the index always starts from 0 and can extend up to the margin of your issue export range. If you want to know the page range and where it ends, you can call the PROJECT.issue_count(jql) method as shown above. Don’t worry, it validates everything, so it'll let you know what you did wrong.

The page arguments help to perform a sort of slicing of the file. Let’s say you downloaded the file at first and the file size is too large i.e. 100Mb or more and you want to break it down. Specifying where the page argument starts and end help in slicing the file into single chunks, so you can do multiples of 10K issues each for example.

The results from the terminal would be shown as well once the script starts to inform you of the progress.

Downloading issue export in CSV format.
<Response [200]> OK ::downloading issues at page: 3 of 6
<Response [200]> OK ::downloading issues at page: 4 of 6
<Response [200]> OK ::downloading issues at page: 5 of 6
<Response [200]> OK ::downloading issues at page: 6 of 6
Processing. Current progress: 25%
Processing. Current progress: 50%
Processing. Current progress: 75%
Processing. Current progress: 100%
Export Completed.File located at /Prince/EXPORT/test_example.csv

Please note as the files are merged into one, the size of the file will increase. I hope this helps to give you the ability to export your projects and migrate them across different instances of Jira and as well help you with having a backup of your entire project(s). 

36 comments

Pramodh M
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
September 5, 2022

Very Insightful, Thanks @Prince Nyeche 

Tiago Almeida September 14, 2022

Thanks a lot for the write up, @Prince Nyeche. This has helped me immensely.

Dania Zaccara October 26, 2022

Hi @Prince Nyeche 

I'm trying your library but I encountered some errors.

I followed the steps:

  1. Created the config.json file with the proper credentials and URL
  2. created the script with a valid JQL:
    from jiraone import LOGIN, issue_export
    import json

    file = "config.json"
    config = json.load(open(file))
    LOGIN(**config)

    jql = "project in (BB) order by created DESC"
    issue_export(jql=jql)

When I run this script with python script.py I get the following errors:

Traceback (most recent call last):
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "script.py", line 9, in <module>
issue_export(jql=jql)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\jiraone\reporting.py", line 1403, in export_issues
"what went wrong with reason: {}".format(reason.json()),
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

 

Python version: 3.7.5
jiraone version: 0.7.2

 

Can you help me please?

 

Thanks,

 

D.

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
October 26, 2022

Hey @Dania Zaccara 

This seems like an authentication issue based on the stack trace on line 1403 which points to the fact that the request returned none. Your first check here will be to confirm if the credentials you're using are valid but the trace, they don't seem to be valid against your instance.

Dania Zaccara October 27, 2022

Hi @Prince Nyeche

thank you for your time!

You were right! I didn't notice a space in the token.

But now I get another error :( 

Downloading issue export in CSV format.
<Response [200]> OK ::downloading issues at page: 0 of 0
Traceback (most recent call last):
File "script.py", line 10, in <module>
issue_export(jql=jql)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\jiraone\reporting.py", line 1585, in export_issues
mark="file", mode="w+")
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\jiraone\reporting.py", line 2523, in file_writer
f.write(content)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u03a3' in position 561: character maps to <undefined>

 

I don't know if it's important, but my JQL returns less or more than 150 issues.

 

Thank you 

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
October 28, 2022

Hey @Dania Zaccara 

The JQL output is not the problem. I've pushed a new version v0.7.3 that should fix the encoding errors in the windows machine, please update your own version. The file-writing mechanism should replace any unknown character that's not UTF-8. Then let me know if that solves things for you.

Dania Zaccara October 31, 2022

Hi @Prince Nyeche

really thanks for your availability.

I reinstalled jiraone and this is the output of the command:

Collecting jiraone
Using cached jiraone-0.7.3-py3-none-any.whl (72 kB)
Requirement already satisfied: requests in c:\users\sardo\appdata\local\programs\python\python37\lib\site-packages (from jiraone) (2.28.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\sardo\appdata\local\programs\python\python37\lib\site-packages (from requests->jiraone) (1.26.12)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\users\sardo\appdata\local\programs\python\python37\lib\site-packages (from requests->jiraone) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\sardo\appdata\local\programs\python\python37\lib\site-packages (from requests->jiraone) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\sardo\appdata\local\programs\python\python37\lib\site-packages (from requests->jiraone) (2022.9.24)
Installing collected packages: jiraone
Successfully installed jiraone-0.7.3

 

But relaunching the script again, this is the result:

Downloading issue export in CSV format.
<Response [200]> OK ::downloading issues at page: 0 of 0
Traceback (most recent call last):
File "script.py", line 10, in <module>
issue_export(jql=jql)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\jiraone\reporting.py", line 1587, in export_issues
mark="file", mode="w+")
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\site-packages\jiraone\reporting.py", line 2531, in file_writer
f.write(content)
File "C:\Users\sardo\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u03a3' in position 561: character maps to <undefined>

 

Am I doing something wrong?

 

Thanks,

 

D.

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
October 31, 2022

No, you're not, it's the character from probably one of the fields within your Jira issues. It is not being appropriately encoded in UTF-8. The file writer converts everything to a string and then attempts an IO process to write it as a plaintext file which it can read later. It's difficult for me to troubleshoot this as I don't use a windows machine, probably if you can help me here by tweaking the code on your local installation on line 1586  by adding this argument errors="replace" as shown below and then rerunning the export with your JQL and let me know if it works then I can upload a new version with the same fix. I'm thinking here that the decoded string contains a character that the file_writer function cannot encode into a text stream.

file_writer(folder, file_name,
content=issues.content.decode('utf-8', errors="replace"),
mark="file", mode="w+")

 You will need to call your python directly after the code tweak, this way it doesn't use your py3.7 version of jiraone. Let me know if you can do the above.

Dania Zaccara November 8, 2022

Hi @Prince Nyeche

I've made the changes but I got the same result.

Thanks to your explanation I understood the problem.

 

First, I made this change:

file_writer(folder, file_name,
content=issues.content.decode('utf-8').encode('cp850','replace').decode('cp850'),
mark="file", mode="w+")

With this, the export ended successfully.

By opening the CSV I saw immediately (looking at the malformed lines) what the problem was.

 

The problem is the attachments!

Looks like not all the attachment types are managed. I discovered that, for example, .jpg is good but PDF is not.

 

Can you please confirm this? And, if you can, can you tell me which attachment types are managed?

 

Thank you for all your help,

 

D.

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
November 15, 2022

Hey @Dania Zaccara 

Please can you add more context here?

The problem is the attachments!


Looks like not all the attachment types are managed. I discovered that, for example, .jpg is good but PDF is not.

Can you share elaborate examples of the field data that doesn't contain any personally identifiable information or mask the data conveying the problem you've seen? Separating the ones marked as good vs not good.

Eric Salenc February 9, 2023

Your jiraone perfectly meets my needs, exporting some jql output in a cvs file.
Nevertheless I cannot authenticate
I'm accessing a Jira server, since LOGIN supports Basic auth only, I wrote:

from jiraone import LOGIN, issue_export, endpoint

jiraConfigSae = "my_windows_login", "my_Base64_encoded_password", "https://rb-tracker.bosch.com/tracker"
jiraConfigSaeMail = "my_email", "my_Base64_encoded_password", "https://rb-tracker.bosch.com/tracker"

jiraConfigurations = [jiraConfigSae, jiraConfigSaeMail]
for jiraConfig in jiraConfigurations:
print(f"\nDBG config: {str(jiraConfig)}")
# change the API attribute to False when
# working with Jira DC or server prior to
# LOGIN instantiation.
LOGIN.api = False

load = LOGIN(*jiraConfig)

# Test my authentication by several basic means
load = LOGIN.get(endpoint.get_all_priorities())
print(f"DBG LOGIN.get:{load.status_code}")
jiraIssueURL = 'https://rb-tracker.bosch.com/tracker/rest/api/2/issue/'
load = LOGIN.get(jiraIssueURL + 'AEEIVGTD-19705')
print(f"DBG LOGIN.get:{load.status_code}")

print('DBG # previous login statement:')
print(LOGIN.get(endpoint.myself()))

req = LOGIN.custom_method('GET', jiraIssueURL + 'AEEIVGTD-19705')
print(req)

Here is the output, I get authentication errors
DBG LOGIN.get:401
DBG LOGIN.get:401
DBG # previous login statement:
<Response [401]>
<Response [401]>

Here is the output, I get authentication errors

DBG LOGIN.get:401
DBG LOGIN.get:401
DBG # previous login statement:
<Response [401]>
<Response [401]>


DBG LOGIN.get:401
DBG LOGIN.get:401
DBG # previous login statement:
<Response [401]>
<Response [401]>

Could you tell me what's wrong in my authentication method?

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
February 9, 2023

Hey @Eric Salenc 

You shouldn't base64 encode the password, it should just be a plain string of your actual password when you're connecting to a Jira Server as the script will generate basic auth for you. Any encoding is handled by the request. 

jiraConfig = "my_windows_login", "my_normal_password", "https://yourinstance.server.com/tracker"
# given that the username correspond to a user in your Jira server
LOGIN.api = False
load = LOGIN(*jiraConfig)

LOGIN also supports the authorization bearer header or any given authorization type in the header. So you can also do

# previous import declaration
# First assign your base_url to the attribute below
LOGIN.base_url = "https://yourinstance.server.com/tracker" token = "GHxxxxxPPPxx" # Your PAT here
LOGIN.api = False # You need to pass the token variable to a keyword argument called `sess` LOGIN.token_session(sess=token) # Afterwards you can make calls such as data = LOGIN.get(endpoint.myself()) 

 Let me know which option works for you.

Like Eric Salenc likes this
Ravneet Narula March 3, 2023
LOGIN(username, password, jira_url)
I am able to login with my credentials.
@Prince Nyeche 
I am not able to export the data, getting below error.

issue_export(jql=jql, final_file=export_file)

 

Traceback (most recent call last):
File "/Users/ravneet/PycharmProjects/pythonProject/venv/lib/python3.8/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 12 column 1 (char 11)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ravneet/PycharmProjects/pythonProject/venv/main.py", line 19, in <module>
issue_export(jql=jql, final_file=export_file)
File "/Users/ravneet/PycharmProjects/pythonProject/venv/lib/python3.8/site-packages/jiraone/reporting.py", line 1405, in export_issues
"what went wrong with reason: {}".format(reason.json()),
File "/Users/ravneet/PycharmProjects/pythonProject/venv/lib/python3.8/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 12 column 1 (char 11)

Process finished with exit code 1

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 3, 2023

Hey @Ravneet Narula it seems that the authentication is failing. For starters, are you connecting to a Jira cloud, Jira Server or DC? If it's the latter, then you need to add the below prior to the login initialization.

LOGIN.api = False
LOGIN(username, password, jira_url)

Try that and see if that works for you.

Ravneet Narula March 4, 2023

@Prince Nyeche 

I am using jira data centre .I am able to login . I have already incorporated this in the script

LOGIN.api = False
LOGIN(username, password, jira_url)

Code breaks at this line : issue_export(jql=jql, final_file=export_file)
and i get the above shared error
Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 4, 2023

Hmm, well I don't think you're being authenticated here with that error. Since you're on DC, you can generate a PAT for authentication. It's a bit different from the usual login mechanism. Generate your PAT and use the below.

from jiraone import LOGIN, endpoint

url = "https://yourserver.datacenter.com"
token = "GHxxxxxPPPxx"
# First assign a base_url to the attribute below
LOGIN.base_url = url
LOGIN.api = False
# You need to pass the token variable
LOGIN.token_session(token=token)
output = LOGIN.get(endpoint.myself())
print(output.json())

Then you can later call the method issue_export

jql = "your jql query here"
export_file = "file.csv"
issue_export(jql=jql, final_file=export_file)

 

Ravneet Narula March 4, 2023

I tried logging in using an API token and the above shared script.Similar error as before.

But it appears to me and you can confirm that i am able to login as i am encountering a different error while fetching issues from a particular project.Snapshot of the code and error received are shared below. I believe i am able to login but whenever i am try to fetch any data from jira it results in an error.Please suggest

 


LOGIN.api = False
LOGIN = LOGIN(username, password, jira_url)
print("Connection established")


project = JIRA.projects()
for project in projects:
issues = JIRA.search_issues('project=XXXX')
for issue in issues:
if str(issue.fields.status) == 'Done' or str(issue.fields.status) == 'Closed':
print (issue)

Error Received:

Connection established
Traceback (most recent call last):
File "/Users/ravneet.narula/PycharmProjects/pythonProject/venv/main.py", line 26, in <module>
project = JIRA.projects()
TypeError: projects() missing 1 required positional argument: 'self'

Process finished with exit code 1


Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 4, 2023

I'm not yet convinced that you are being authenticated based on the error shared, it shows that the failure occurs because the status_code is greater than 300. How about starting from the beginning all over?

A simple auth method just to confirm what you get

# previous statements or declarations
LOGIN.api = False
LOGIN = LOGIN(username, password, jira_url)
output = LOGIN.get(endpoint.myself())
print(output.status_code, output.reason)
print(output.json())

If there are no errors and shows less than 300 status codes from the above that means that your auth method or credentials are valid and you'll get some output from your Jira DC. However, if that's not the case, then you should get a reason why the authentication failed. Try that first to see what's there.

Ravneet Narula March 6, 2023

 

I checked with shared code.I am getting the below shared error

 

Traceback (most recent call last):
File "/Users/ravneet.narula/PycharmProjects/pythonProject/venv/main.py", line 25, in <module>
output = LOGIN.get(endpoint.myself())
AttributeError: 'NoneType' object has no attribute 'get'

Process finished with exit code 1

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 6, 2023

Try running the script from your command-line tool for python3. It looks like it's calling another object and not the one in context.

Ravneet Narula March 8, 2023

@Prince Nyeche I tried running the script from terminal .I got the same error .Error shared below:

 

Traceback (most recent call last):

  File "/Users/ravneet.narula/PycharmProjects/pythonProject/venv/main.py", line 25, in <module>

    output = LOGIN.get(endpoint.myself())

AttributeError: 'NoneType' object has no attribute 'get'

ravneet.narula@ravneet venv % 

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
March 8, 2023

Hey, @Ravneet Narula please open an issue here so I can follow up from there.

Madhav Dube April 5, 2023

Is there a way to get the export with limited number of fields? @Prince Nyeche 

Prince Nyeche
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
April 5, 2023

Hey @Madhav Dube - As of now, no but hopefully I'll be including that in the next version update v0.7.6. There are two ways for this: export default fields, the same as shown on the UI, and the second is, field exclusion, which allows only fields specified in the argument to remain after the export is completed. In addition, the other part which I thought is necessary is a JSON export, so in a near future update, a user can use this method to choose between "CSV or JSON" as an export. I'm trying to work out the easiest way to make that happen with this same method to help Jira admins better. 

Madhav Dube April 5, 2023

Thank @Prince Nyeche when you say export default fields. Is it what exists and can use the existing export_issues function or have to wait for you to release the next version?

Comment

Log in or Sign up to comment
TAGS
AUG Leaders

Atlassian Community Events