Python REST API search_issues call with pagination is returning duplicate records

[Sanjay Kinger]

I have implemented pagination call as mentioned in the documentation. Script is working fine except with a major glitch that it is returning few duplicate records and hence failing to return few records. Here is the pseudocode

1. LIMIT=100
2. Totalissues = search_issues(query,MaxResults=0)
3. For start in range(0, TotalIssues, LIMIT)
4.     Issues= search_issues(query, startAt=start,MaxResults=LIMIT)

This is returning duplicate issues across different pages. E.g.

If TotalIssues = 1200, an issue 230(just for example) is returned in page 3 and page 4 calls.surprisingly after iterating through all pages total issues returned are 1200. But due to duplicate issues, some actual results are NOT returned by API.

Moreover, number of duplicate records vary with change in LIMIT value. Lower the value larger are the duplicate records. 

Has anyone observed such issue?

[DPKJ]

@Sanjay Kingerhave your tried accessing these rest api by any other means,

• Atlassian Rest API Browser(only available for server) or
• PostMan, or any other browser tool

As this will help us identify the problem is with Python Library or Jira itself.

[Sanjay Kinger]

Could not reproduce issue with PostMan.

Morever, issue is not consistent. Sometime I get duplicate from page 0 or some time page 1. And the number of duplicate records change everytime. 

For 2000+ total issues, fetched in bunch of 500, approx. 300+ duplicates are reported. This completely skews my report. 

08/27/2019 05:15:25: INFO: Processing JIRA data (500 of 2079)
08/27/2019 05:17:03: INFO: Processing JIRA data (1000 of 2079)
Duplicate issue QVS-9381 reported from page 0
Duplicate issue QVS-9375 reported from page 0
Duplicate issue QVS-9379 reported from page 0
Duplicate issue QVS-9376 reported from page 0
.

.
.
Duplicate issue QVS-8274 reported from page 0
Duplicate issue QVS-8313 reported from page 0
08/27/2019 05:18:56: INFO: Processing JIRA data (1500 of 2079)
08/27/2019 05:20:40: INFO: Processing JIRA data (2000 of 2079)

[Sanjay Kinger]

Even the number of duplicates change from run to run - sometime 100, sometime 300

I have introduced delay of 15 seconds between two consecutive calls. Once I got no duplicates, but randomly in some runs I'm still getting duplicates. 

Number of duplicates is also changing based on batch size. smaller batch size, more duplicates.

[Sanjay Kinger]

Anyone else facing similar issue or knows a solution? 

[Elliott Herz]

My initial hunch is that you don't have an order by clause in your search JQL.

My query is just 'ORDER BY created ASC', and that seems to work perfectly.

[DPKJ]

@Sanjay Kinger  I agree with @Elliott Herz  this can be a issue of ordering. Try adding a `Order By` clause.

[Yuvam Jain]

I also faced this issue and by adding Order by created in my JQL, I am getting all unique records. It means that, Order By is now mandatory to get proper results.

Any clue why it was happening ?