Setting up REST Pagination in Azure Data factory for Jira Align

Mark Northcott_clxwl6

Hello All,

I'm trying to setup pagination in the the CopyData Source tab in Azure Data factory but nothing seems to work. 

I understand top and skip values are the JA alternatives to maxresults and startat but I don't understand how to apply them. I don't seem to have a next page URL. Nothing else seems to work. 

There has to be a simple method right? All I want to do is collect the full recordset, not the first 100, and add them to a SQL Table. 

Allan Maxwell

Here is an example using orderby, top and skip.  This call is repeated in a loop until the resultset contains less than the $TOP value, indicating you have reached the end of the virtual resultset.

• $TOP set the number of records to return in each call which will typically be 100 , the maximum.
• $SKIP will be programmatically incremented, typically be the $TOP value.  The first call $TOP = 0, second call $TOP = 100, third $TOP = 200, ect.
• It is very important that the GET product the same resultset each time you call it, or as close as you can get it in the absence of true read-consistent transaction management.  I typically order what I assume would be the primary key for that data.  In most cases this is going to be the ID field.

 

GET https://YOURCOMPANY.jiraalign.com/rest/align/api/2/Epics?$select=id,title&$filter=primaryProgramId EQ 19 &orderby=id DESC &$top=100&$skip=0