startAt and maxResults race condition

James Brown January 20, 2016

Hi,

I have a program that uses the REST API to take snapshots of my team's issues a couple of times a day. A snapshot would usually contain several thousand issues.

A snapshot is generated by making a multiple calls to the REST API using JQL, incrementing the startAt value each time by the number of issues returned in the previous call. The program stops when the 'total' number of issues (read from the first response) have been received.

There appears to be a race condition introduced by this iterative process that can cause the program to run forever, or in fact introduce other inconsistencies into the snapshot.

The issue is most easily understood as being a search for all open issues, where a number of issues become closed between reading the initial total and a subsequent attempt to retrieve all open issues. Because some issues have been closed, the attempt to read the initial total will never complete.

Is there a way to handle this race condition?

3 answers

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

Post a new question

1 vote
MattS
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

I have not had this problem when using a fixed block size. This is how I use it:

 

block_size = 900 # 1000 is the default maximum
    block_num = 0
    while True:
        start_idx = block_num*block_size
        issues = jira.search_issues(jql, start_idx, block_size)
        if len(issues) == 0:
            # Retrieve issues until there are no more to come
            break
        block_num += 1
        for issue in issues:
            log.info('%s: %s' % (issue.key, issue.fields.summary))
Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

+1

0 votes
James Brown January 21, 2016

We use them for various programmatic tasks

Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

Can't you just use a timeout thing in order to avoid it?

Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

You could even set the timeout amount of time proportional to the total number of issues it has to process.

James Brown January 21, 2016

Breaking out of the loop isn't really the problem. It's more the fact that each request is performing a new search and returning the nth page. If anyone updates a JIRA from one request to another in a way that influences the order/number of tickets returned by the search, we end up with either missing or repeated JIRA issues in the assemblage of all the request results. What we want to do is get the results of all the requests as a transaction, or using an asoftime, or get them all in one go without having to iterate, or similar...

Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

...or keep track of the last issue processed and if its key doesn't change to a different one in the following 10 seconds (for example), then finish the iteration.

Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 21, 2016

Ok, I understand.

Wouldn't it be good to subscribe to that search to receive the snapshot of the issues twice/day by email and parse its content?

The email can be triggered by clicking on a button as well...

0 votes
Ignacio Pulgar
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
January 20, 2016

What is the purpose for collecting those snapshots?

If it is just for keeping them on a history of archived issues, then I would suggest using filter subscriptions instead, so that you are sent by email a snapshot with the issues and columns you want to store.

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

Post a new question

TAGS
AUG Leaders

Atlassian Community Events