Need to extract data from Jira

Dave July 24, 2013

Is there a way to extract all the data for a project from an external script?

I need run a job that daily extracts Jira data for a set of projects. Is there a way to extract data on a per project basis with an external URL. (Say in XML format)?

thanks

Dave

7 answers

2 votes
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
July 25, 2013

Hmm. It takes a long time and breaks because it's kind of the wrong approach - Jira simply isn't designed for massive data dumps, and doing it by extract (whether it's excel, xml, rest filters etc), you're going to run into performance issues, and it's very questionable practice demanding vast amounts of data for "dashboard" data analysis when you're probably realistically dumping 99.999% of it. For example, if you need to know "how many issues are open in project X", then that's a filter statistic, which Jira has right in hand, irrespective of the size of the data. Downloading all the details of every issue is a vast amount of data, just to re-count them.

I'd take two steps here.

First, see if Jira can tell you what you need to know without downloads. I'd look at the dashboard gadgets. As an example, you can have a filter for "all issues in Jira", and use that in a dashboard gadget like "filter statistics" - that could tell you how many you've got per project instantly. Or by type, or status etc. It doesn't grab any data from the filter, the numbers are in there already. Then try the 2d stats - I find that one a LOT more powerful - you can do "project vs status" for example...

Anyway, I don't know your data, but I'm suggesting you do a spot of analysis. Or, heck, tell us what your dashboard needs and we can try to point you at something (or do the usual sharp-intake-of-breath plus we-need-to-think-about-it)

Secondly, if Jira is capable of generating something useful for you, it's getting it into the other app - does it support OSGI gadgets? If so, you can just drop Jira dashboard gadgets into it! Or, going back to your original question, it could be persuaded to scrape the results. You could look at writing a simple Jira report designed to be scraped, and so-on...

1 vote
Midori
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
July 18, 2016

If the Excel file formats (XLSX, XLSM or XLS) or plain vanilla CSV files can be imported to the target system, then make sure you give the Excel Automation Plugin a deep look.

This could save us lots of work: it can periodically execute JQL searches, transform their results to customizable Excel output and write the result to the filesystem. You could then configure the other side to pick up the latest file from that location.

Roopa Raghavan March 25, 2019

Ferenc - thats a great way to summarize. import excel or CSV file into target system and analyze and configure target to pick up latest file from that location. On paper, this would be a great solution for my needs. I'd have to implement and see if it works for our Jira cloud implementation.

Thanks again.

 

0 votes
Alex Perez
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
July 25, 2013

Dave,

That-s why I recommended you to use the backup as XML feature.

REST api (AFAIK) is not intended to this type of operations, it will help you to know details of one issue, but not to do a massive dump.

This use case sounds to me as dataware house operations. You're planning to refresh once a day, so I guess that your users can tolerate to have reports with "yesterdays backup", and it's unnecesary calculate all reports with live data, thus not degrading productive environment.

I suggest you to dump jira contents (as XML backup or a database export/import) daily, and then calculate your reports over the copied data (custom program to parse XML or SQLs), then send the info to your dashboarding system

HTH

0 votes
Dave July 24, 2013

We have a separate dashboarding system that extracts data from various repositories across the enterprise and centralizes reporting. We need the reports to appear in the dashboard rather than individual systems.

Right now we have a filter that dumps the entire Jira database via 1 filter (for all projects). However it takes a very long time and is prone to breaking. I would like to keep an external list of projects we need and then extract 1 project at a time.

So it sounds like this is not possible?

0 votes
Dave July 24, 2013


In this particular case, defining a filter isn't practical. There is a large list of projects to export and this list changes over time. Is REST the best way to extact a project-by-project dump of data?

(I am looking at the REST documentation but it is not very clear)

Dave

Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
July 24, 2013

Well, REST will let you code more dynamic stuff, but it is still going to rely on you saying "run a filter for project X and then do stuff with the results"

I think it might be worth asking what you're doing with this large extract - what's it for, why is it so large and why can't you do the work in Jira?

0 votes
Nic Brough -Adaptavist-
Community Leader
Community Leader
Community Leaders are connectors, ambassadors, and mentors. On the online community, they serve as thought leaders, product experts, and moderators.
July 24, 2013

Definte a filter for "all issues in my project" and run it, so you get the list in the issue navigator.

Then, look to the top right for views, and look on the list for "xml". Right click and copy the url.

Anything hitting that url (curl, wget) will be handed the results as xml.

It's more elegant to do it in REST of course - lots more powerful because you can define the filters etc. But raw xml for a fixed project is a start.

0 votes
Alex Perez
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
July 24, 2013

maybe a complete backup as XML jobs can help you?

exploring the structure perhaps you can filter by project ..

HTH

Suggest an answer

Log in or Sign up to answer