Jira and Jira Align Integration: Sync Time Throughput and Connector Performance

[#1]

Jira Align runs 4 different processes to sync data between Jira and Jira Align:

1. Searching Jira for newly created and updated Fix Versions and Issues: 

Example API calls from this Process:

/rest/api/2/project{/ID}
/rest/api/2/project/{ID}?{QUERY}
/rest/api/2/version/{ID}
/rest/api/2/search

This process will query Jira using Post and Get API calls for each integrated Jira Project and integrated Jira Issue Type combination.  Under Jira Align Admin > Jira Settings > Jira Setup > Timer on Jira Issues connector admins will define the break in minutes the Jira Align connector takes between each cycle of Project and Issue Type combinations.  

If the Timer on Jira Issues Timer is set to 7 minutes and it takes the connector 20 minutes to run through the process, any given issue created or updated in Jira will sync to Jira Align in somewhere between 1 minute and 27 minutes depending on where the connector is in its current process.  Our larger customers with 500 plus integrated Projects will see 20 to 45 minute cycle times here.  Our smaller customers with dozens of projects integrated will see 2 to 5 minute cycle times here.  Cycle times will be impacted by the amount of API calls sent between Jira and Jira Align which is a direct correlation tied to the number of integrated Projects and the number of integrated custom Issue Types, the response time from Jira on API calls, and the amount of parallel API calls being ran by the other processes or other plugins.

2. Continuously creating new Issues and updating existing Issues in Jira from Jira Align:

Example API call from this Process:

/rest/api/2/issue

This process will sync newly created work items and updated work items from Jira Align to Jira.  Under Jira Align Admin > Jira Settings > Jira Setup > Timer on Continuous Push to Jira connector admins will define the break in minutes the Jira Align connector takes between each cycle of syncing updates from Jira Align to Jira.

We typically recommend setting the break timer for Timer on Continuous Push to Jira to be between 1 and 7 minutes.  Our smaller customers will see cycle times in the 1 minute range for this process and our larger customers with upwards of 500 integrated Projects will see cycle times closer to 5 to 6 minutes.  We are in the process of optimizing how Jira Align queries its database for work items that need to sync to Jira and we are seeing early promising results.  It should be expected that issues will sync from Jira Align to Jira within seconds to 10 minutes depending on the amount of integrated Projects and parallel API calls. Customers with only a couple dozen integrated Projects should see work items sync from Jira Align to Jira in less than 5 minutes almost all the time barring a sync error. 

3. Syncing Boards and Sprints from Jira to Jira Align:

Example API call from this Process:

/rest/agile/1.0/board/{ID}
/rest/agile/1.0/board/{ID}configuration
/rest/api/2/filter/{ID}
/rest/agile/1.0/board/{ID}/sprint?{QUERY}

This process will query Jira for integrated boards, newly created sprints, and any changes to the status or dates of integrated sprints.  Under Jira Align Admin > Jira Settings > Jira Setup > Timer on Jira Boards connector admins will define the break in minutes the Jira Align connector takes between each cycle of querying the Jira Boards and sprints.  

We typically recommend setting the Timer on Jira Boards to 1 to 5 minutes during the initial on-boarding of a connector and testing phase to rapidly map new boards and sprints.  Once a connector is in a stabilized form or maintenance mode we recommend setting the Timer on Jira Boards to 30 minutes for customers with less than 50 integrated boards, 2 hours for customers with between 50 and 250 boards, 4 plus hours for customers with greater than 250 boards.  The board sync is an expensive process and can lead to the other processes taking longer.  Our customers will typically increase the Timer on Jira Boards break time in order to reduce the total number sent from Jira Align to Jira.  Our smaller customers will see a board sync cycle run in 1 to 5 minutes while our larger customers will experience board sync cycles of 1 to 2 hours.

Notes: Currently new teams are synced with Jira Align near the end of the subsequent Board Sync process.  The time it takes a newly mapped board to sync to a Jira Align team will be the time for the current process to complete, plus the time of the connector's Timer on Jira Boards break, plus the time of the subsequent new Board Sync process.  We are working on updating the custom JQL functionality to allow Admins to query newly integrated boards to more quickly sync the teams.

4. Custom JQLs executed in Jira Align, the Jira Deleted Issue Report, and custom Jobs such as pull rank and resync configurations:

Example API calls from this Process:

/rest/api/2/search

The Jira deleted issue report which is ran under Jira Align Admin > Jira Management > Jira Deleted Issues will search Jira for the issues that correspond to non Accepted or Closed work items in Jira Align.  Any issues that the connectors finds to have been deleted in Jira will show on a list in Jira Align for connector admins to ignore, disassociate the Jira Align work items from Jira, or to hard delete the work items.  This process is extremely expensive as it will send API calls searching for thousands of Jira issues so it should be scheduled for early on Saturday mornings to avoid overloading Jira or slowing down other processes.  

Custom JQLs are ran under Jira Align Admin > Jira Settings > Jira Integration.  We recommend scoping the JQLs to specific Projects, setting the Board Sync configurations to No, and running this custom queries later at night or earlier in the morning to avoid the slow down of other sync processes.  We go into further detail on Custom JQLs and performance recommendations for them here: https://community.atlassian.com/t5/Jira-Align-articles/Jira-and-Jira-Align-Integration-Jira-Query-Language-JQL-Best/ba-p/1260051

The pull and push rank functionality is described here: https://help.jiraalign.com/hc/en-us/articles/115002917308-10X-Prioritize-Rank-Work-Items-in-the-Backlog The pull and push rank process will be placed into a queue to be ran after the ongoing connector sync processes so we would expect this process to take a time-frame tied to an average value of the other processes.

The resync configurations process which can be used to re-pull statuses and customer fields from Jira takes about 30 seconds to run, but will be queued in a similar fashion to the push and pull rank process.

[Mark Cruth]

Excellent article on integration with some great best practices!

[Arden_Hawley]

Nice writeup Tim!

[Kent Baxley]

Excellent writeup!  Demystifies a lot of what goes on with the timers and will go a long way toward helping customers tune their environments. 

[Richard Wilson _Jarow Digital_]

Thanks a whole heap @Tim Keyes Very useful to know and understand better.

[Vijay Byndoor]

Thanks @Tim Keyes ,

This is good information on the underlying working of the connector. As per our other discussions, can we get an understanding of the upcoming new connector and how it aims to solve specific sync delay issues due to a big number of issues and thus longer cycle times for the sync.

[Tim Keyes]

Hi @Vijay Byndoor

Updated versions of the connector are being worked on to leverage the relationship of Jira and Jira Align within Atlassian.  The updated connector is being designed to work on activity logs to transmit updates as opposed to wider external searches which should yield significant improvements to sync throughput.  We are still working through timelines on the enhancements.

Cheers!
Tim

[Carlos Garcia Navarro]

Nice article, thanks for sharing!