Extracting complete issue history via API

[#1]

There are a lot of things that can be said when you have a history of such information. Given that Jira is a great piece of software, it provides a means whereby there's data integrity when something is changed within a project. The issue history feature by far is one of the most vital features that Jira has to offer. The complexity involved in recording data and the changes that took place makes it an integral part that shouldn't be easily manipulated.

Hence, I believe one of the fundamental things to keep in mind while moving from server to cloud is the ability to retain issue history. However this is a very complex process to do and in most cases, you end up not transferring this data due to the complexity involved.

Alternatively, it doesn't mean you can't extract such information from either a server to cloud instance or vice versa or even a cloud to cloud instance of Jira. The way to go about it, is to use Jira's REST API and the end result is an output in a CSV file which can be viewed from any spreadsheet application.

Most of the API related instructions here are based on python examples using a specific package called jiraone (can be installed using pip install jiraone). Which can simply output a report file in CSV format. Example for Jira server instance given below

from jiraone import LOGIN, PROJECT

user = "username"
password = "password"
link = "https://yourinstance.atlassian.net"
LOGIN.api = False  # comment out line, if you want to extract history from a cloud instance
LOGIN(user=user, password=password, url=link)

if __name__ == '__main__':
    # the output of the file would be absolute to the directory where this python file is being executed from
    jql = "project in (PYT) ORDER BY Rank DESC"  # A valid JQL query
    PROJECT.change_log(jql=jql)

With the above, you will get a complete extract of all issue history in CSV format within the specified project listed in the JQL variable.

[M Amine]

Very interesting content

[Tristan Munthree]

Hi Prince,

I have used this code and it runs well and gets the history of issues but at a certain point I get the error: "expecting value line 1 column 1 (char 0)" which is when the code stops running. Can I please get some help on how to go about fixing this error so that it does not interrupt the code from getting all of the issue history? Your assistance will be appreciated.

[Prince Nyeche]

Hi @Tristan Munthree 

Please can you create an issue on the package in Github with the stack trace error, so I can trace where the error is coming from?

[Tristan Munthree]

Hi @Prince Nyeche 

I have just created an issue with attachments showing the error, thank you for your prompt response.

[Miguel Angel Velazquez Solis]

Where should I run the script?

Can I do it directly on ScriptRunner?

[Prince Nyeche]

No, this is a python script. You will need to install python on your device and run it from your terminal or if you have an IDE, you could run it from there too. 

[Miguel Angel Velazquez Solis]

@Prince Nyeche 

Thanks for your comments, I have tested the script and it works but does it have export limits?
I have to export the information of more than 1000 records, is this possible?
What do I have to modify in the code or where can I consult more information to achieve this?

[Prince Nyeche]

Yes, there's no export limit. You can go to the doc https://jiraone.readthedocs.io

[Miguel Angel Velazquez Solis]

Hi @Prince Nyeche

When I run the code I get the error below

UnicodeEncodeError: 'charmap' codec can't encode character '\x95' in position 1114: character maps to <undefined>

I checked the issue and this happens because the custom field (objective) has bullets as example below

Do you know how can I solve this?

[Prince Nyeche]

Hey @Miguel Angel Velazquez Solis 

It's suppose to try to convert that using utf-8 character encoding as that's the default on unix-like systems. However, I think if you're using windows this has to be explicitly mentioned when writing to a file. I'll push an update about this problem. Please can you create an issue here with the error that you received and I'll follow up with a resolution in the next update.

[Miguel Angel Velazquez Solis]

Hi @Prince Nyeche

Thank you so much 

I have already created the ticket in github.

Regards

[Andrew Rainboldt]

Hi,

How are credentials passed to the Project object?

The code above uses the `token_session` to initialize the `auth_request` attribute of the LOGIN class. But inside the PROJECT class the LOGIN class isn't ever initialized and no parameters are passed from the instantiated LOGIN class to the environment or PROJECT class in any way. 

How is this supposed to work?

I'm getting this error:

JiraOneErrors: <JiraOneError: Authentication failed. Please check your credentials.>
 

Thanks,
Andrew

[Prince Nyeche]

Hey Andrew,

Great question. To get the authentication to work, you need to instantiate the LOGIN  alias or class once hence the call

# previous login variables
LOGIN(user=user, password=password, url=link)

Once that is done, you can access all the objects accessible to that class from anywhere within the code. The LOGIN alias is a global object for authentication that is used with the classmethod for Jira's API url called endpoint. So since the LOGIN performs authentication and HTTP requests, we can do something like after it's instantiated.

LOGIN.get()

And since this is global when combined with the classmethod for each endpoint alias or class. This passes the entire session plus the endpoint class to the HTTP methods extracting the API contents. The PROJECT alias mostly being static doesn't really need instantiation. However, note that it is already instantiated automatically hence it can be called directly. With the above when combined together, each session is passed sequentially and the session lives in memory until the script is terminated or you terminate the connection or whatever objects you've called. Now, coming to why you get the error, there could be a number of reasons but the only thing that can cause that error is either a wrong email address or token (if on Jira cloud) or a wrong username or password (if on Jira server or DC).

If you're on Jira server or DC and using PAT, I think this might not work as I didn't test such authentication mechanism. So it's best to stick with username and password. OAuth 2.0 works as well but it was only tested on Jira cloud, so its mechanism might not work for Jira server or DC. Let me know if that answered your question.

[Andrew Rainboldt]

Hi, 

Thanks for the quick response. Now I see the way LOGIN is created, via this

LOGIN = InitProcess()  

 and then re-instantiated when called. 

Thanks for the explanation.

Regarding the error, I'm just trying to access my company jira server with username and password as indicated in the example. 

I'm running out of Jupyter, perhaps something odd is happening there, but l've switched to cmd and the same error is coming up

Thanks

[Prince Nyeche]

Before calling the LOGIN alias since you're connected to a Jira server instance do the below

# previous login variables
LOGIN.api = False
LOGIN(user=user, password=password, url=link)

As Jira server still uses v2 of Atlassian API. The attribute `LOGIN.api` changes the endpoints from api `v3` which works on cloud to `latest` which works on Jira server or DC. Probably that's why the auth keeps failing because it cannot connect at all.

[Andrew Rainboldt]

Hey Prince,

thanks, authentication still doesn't work. It could be to do with the way my company manages accounts. 

Thanks anyway

[Agnès Masson-Sibut]

Hello there,

I managed to connect to my company Jira Cloud instance using the API token (https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) but I still have the same error than Miguel:

UnicodeEncodeError: 'ascii' codec can't encode character '\u0117' in position 94: ordinal not in range(128)

Did you find any workaround for that @Miguel Angel Velazquez Solis ?

Thanks

Regards 

[Prince Nyeche]

Hey, @Agnès Masson-Sibut  You should download the new version v0.7.3 which should fix the exception error of encoded characters.

[Michael Walker]

@Agnès Masson-Sibut can you put the code you used to connect with the api token?

[Handy Rey]

Hello @Prince Nyeche

Is there a way to query more specific filter?

someting like "project = <your_project> AND issuetype = Story AND status = Done order by created DESC". I've tried JQL syntax above, it returned an error.

Thank you.

[Prince Nyeche]

Hey @Handy Rey 

Any valid JQL should simply just work, except it returned no issue. However, to target a specific field, you need to use the field_name argument. For example

# previous expression/statement
field = "Summary"
PROJECT.change_log(jql=jql, field_name=field)

That way, you export only the values from that field if it exists as part of the history. The name of the field has to be an exact string of the field name, the way it is represented on Jira.

[Alexander Scholz]

Hey @AlL, 

we were searching for a solution to exctract the full history of all issues via REST API to a Qlik Cloud project. 

Since there is no possibility, here is a ticket to implement the function. 

Ability to extract issue history of all issues via API 

Please vote. :-)

Thanks

[Prince Nyeche]

Hey folks, the latest version of jiraone comes with an asynchronous function to extract the issue history very fast. Example

# import statement
PROJECT.async_change_log(
               jql, folder="TEST", file="sample.csv", workers=20, flush=10
            )

An explanation of the arguments can be found here

[Camilo A. Soto]

Using atlassian-python-api==3.41.11 you can query the change logs 

from atlassian import ServiceDesk, Jira
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

jira_url = ""
jira_token=""
jira = Jira(url=jira_url, token=jira_token, verify_ssl=False)
changes = jira.get_issue_changelog("1982769")
for history in changes["histories"]:
  for item in history["items"]:
    print(item)
  print("========================")