Setting up REST Pagination in Azure Data factory for Jira Align

Mark Northcott October 16, 2022

Hello All,

I'm trying to setup pagination in the the CopyData Source tab in Azure Data factory but nothing seems to work. 

I understand top and skip values are the JA alternatives to maxresults and startat but I don't understand how to apply them. I don't seem to have a next page URL. Nothing else seems to work. 

There has to be a simple method right? All I want to do is collect the full recordset, not the first 100, and add them to a SQL Table. 

1 answer

0 votes
Allan Maxwell
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 16, 2022

Here is an example using orderby, top and skip.  This call is repeated in a loop until the resultset contains less than the $TOP value, indicating you have reached the end of the virtual resultset.

  • $TOP set the number of records to return in each call which will typically be 100 , the maximum.
  • $SKIP will be programmatically incremented, typically be the $TOP value.  The first call $TOP = 0, second call $TOP = 100, third $TOP = 200, ect.
  • It is very important that the GET product the same resultset each time you call it, or as close as you can get it in the absence of true read-consistent transaction management.  I typically order what I assume would be the primary key for that data.  In most cases this is going to be the ID field.

 

GET https://YOURCOMPANY.jiraalign.com/rest/align/api/2/Epics?$select=id,title&$filter=primaryProgramId EQ 19 &orderby=id DESC &$top=100&$skip=0

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events