Issue with Data Inconsistency in nextPageToken-based Pagination for Updated Issues

AutomationSyncUser

Description:

This is regarding the recent deprecation of the old Jira issue search API and the shift to the new nextPageToken-based pagination mechanism. Referring to: Atlassian developer changelog

I'm facing a critical issue when fetching issues between two timestamps, say T1 and T2, using the new search API. The problem arises during pagination:

1. I fetch the first page of results between T1 and T2.

2. Before fetching the second page, an entity from the first page is updated, causing it to no longer meet the original query filter (i.e., it now falls outside T1–T2).

3. This update causes the remaining entities to shift between pages, resulting in data loss or duplication, as the nextPageToken does not guarantee a snapshot-stable view.

Constraints I'm facing:

• Jira limits the number of issues I can fetch in one query to 5000 issues.

• In contrast, I can bulk update up to 1000 issues at once. Additionally, at any given time, only 5 concurrent requests can be sent across all users. So, I can update issues faster than I can query them.  Referring to: The Jira Cloud platform REST API

What I'm Looking For

Is there a way to:

1. Reliably fetch all issues updated between T1 and T2, without missing any due to shifting caused by mid-fetch updates?

2. Get all issues that had at least one revision in a given timeframe?

3. Anchor a result set (snapshot or revision token) so pagination remains stable?

unknown

Hello @AutomationSyncUser 

You seem to have fundamentally mis-understood the very nature of the data you are trying to obtain. The issue data in Jira isn't static... it's dynamic. All the things you are observing in the behaviour of the REST API and the pagination token are a direct result of that... and the expected behaviour.

It's not possible to extract such data over such a large number of dynamic issues and expect that not one single change will happen to any of those issues between the time you start your query and the time you finish. You seem to be somehow expecting or wanting all those issues or their data to enter some sort of 'frozen' state while you are in the process of executing your query.

Over the scope of the vast numbers of issues you are describing, of course some of those issues are going to change, or be increased in number, or decreased (IE, deleted), while your query is 'mid-fetch', as you call it, so you can't expect the pagination to also be 'stable', as you call it, at the same time.

When you make a query, with JQL for example, the results returned represent a set issues that match the query or data state at that exact point in time. That full set of results is cached and you iterate the set of results, page by page, but while you are doing that, those issues are still dynamically changing in the background, so you could run the exact same query only moments later, and get very different results, with a different total count, and even a different number of pages to contain that different total count.

This basic concept also applies to those API endpoints using the other form of pagination that use startAt and isLast parameters

You really should discuss these topics with the programmers and developers at your organisation to get a better understanding of what you're dealing with.