curl versus Python versus Export: no simple solution?

Question

I need to connect Jira data to an external data source to get a complete picture of all the efforts underway.  I have tried multiple solutions just to get the Jira data, each ending up overly complicated.

* Build a filter & dashboard, then export to CSV - I can create a dashboard with the records and columns that I want, but I can only export 1,000 records at a time.  Given that there can be over 10,000, this is too manual to do on a routine basis as the filtering to break it into small bits is too cumbersome.
* I tried to read the data via Python.  No matter what I try, though, I get the dreaded SSL error "max retries exceeded" that I've seen posted in a variety of places.  None of those solutions ever eliminates the error.
* I've successfully connected and retrieved information via curl, saved the resulting response to a json file then read it into Python ... sort of.  The resulting data gives way too many columns which are mostly empty and several columns yield a dictionary within a dictionary and sometimes within a dictionary.  This is in contrast to the various curl examples I've seen where the example shows data that is fine for an example: less than 5 records with less than 10 columns and no nested dictionaries.  I've been able to slowly build out a curl that limits the data but is still no where near as simple as the CSV export I can get with #1.

I don't mind using curl to retrieve the data but it would be nice to just keep everything in Python.  Unfortunately the system admins aren't expert in the security intricacies so they pointed me here, to the Atlassian community boards.  If I have to use curl, that's fine, but I find I'm building it out by trial and error by googling a question, trying something, googling a different question, getting more info and refining, repeat until done ... which seems a long way off.  Even then, I'll probably have to do the same thing in Python to read the JSON output.

There has got to be a better way, I hope.  If it helps, below is the curl request I've built up so far (minus anything proprietary, of course).  Any wisdom or advice is greatly appreciated.

curl -D- -H "Authorization: Basic xxxxxxxxxxxxxx==" -X POST -d "{\"jql\": \"'project' = 'MYPROJ' AND 'issuetype' = 'Milestone'\",\"fields\":[\"key\"]}" -H "Content-Type: application/json" --url "https://theprocess.thecompany.com/rest/api/2/search"

Prince Nyeche - ELFAPP · Answer

Just thinking about this, I came up with this script as an example which is completely expandable based on what fields you want.

Prince Nyeche - ELFAPP · Answer

You have to build your output data structure first if you're going to connect Jira to an external data source. If you want a flat CSV file, you will need to know the exact fields you want to be placed on that file. An easy way to do this in python would be

* Define your output data structure. i.e. name of fields you want present.
* Run a JQL API search for the issues and retrieve the payload from each issue of the search.
* Curate this list of data from each issue and add them into the above structure. i.e. putting it in a list is better in an ordered sequence.
* After the search is complete, sort the data (if needed) and send it as output to create the data into a CSV file as needed.

Nic Brough -Adaptavist- · Answer

Have you tried adding the fields parameter as described in the first link I gave?

It will still return the metadata in the JSON, but you can easily trim that out in your code.  I often work with shell scripts rather than python, using a simple executable for extracting raw data from json responses - it's called "jq" and I understand there's a library for python that does the same thing (pip install jq)