Optimizing Data Retrieval with /rest/api/3/search/jql Pagination

Question

As I transition from the soon-to-be-deprecated "/rest/api/3/search" endpoint to the new "/rest/api/3/search/jql" endpoint, I have encountered a challenge related to pagination and its impact on performance. I am seeking advice on optimizing our approach to minimize the delay in data retrieval.

Previously, using "/rest/api/3/search", I followed this approach to efficiently retrieve all issues:

* Fetch the total number of issues.
* Calculate the number of required requests.
* Use the startAt parameter to control pagination.
* Execute multiple batch requests concurrently and resolve them together, significantly improving performance.

However, with the new "/rest/api/3/search/jql" endpoint, the response structure has changed:

{ "issues": [...], "nextPageToken": "..." }

Since startAt is no longer available, the only way to paginate is by using nextPageToken, which is dynamically generated for each request. As a result, subsequent requests must wait for the previous response to obtain the required token, preventing parallel execution and significantly slowing down the data retrieval process.

Given this limitation, we would appreciate any guidance on best practices for handling pagination efficiently with this new API. Are there any alternative approaches or optimizations that could help mitigate the impact of sequential fetching on performance?

Merchant, Craig · Answer

nextPageToken doesn't even work.  It's completely broken.  If you put in the token returned from the first request, it returns the first page again.

Infinite loop.

Unbelievable that Atlassian replaced a perfectly good API endpoint with a completely broken one.

Akash Singh_vujl0e · Answer

@Olha Pruska Welcome to Atlassian Community!

To add to Aron's answer, the only available option at the moment is to limit the fields being fetched by specifying only the necessary fields in your request by using fields parameter. This allows you to increase the maxResults value, which can go up to 5000 if fewer fields are requested. This way, the total number of sequential requests can be reduced.

Aron Gombas _Midori_ · Answer

As far as I know there is no alternative to "sequential paging".

When Atlassian announced the change, one of their reasons was that it will make it harder for "aggressive" apps to overload the Jira Cloud REST API and therefore to endanger the platform's heath.