Github - Sync Paused: GitHub rate limit was exceeded.

[Jakob KN]

Hi folks!

We have a Jira Data Center (7.9.2) instance where our Github repositories are connected via DVCS account.

It seems we've started to encounter failures in sync of our repositories, due to the sync limit imposed by Github for DVCS accounts (we have 725 repositories connected).
The following message appears for some repositories:
Sync Paused: GitHub rate limit was exceeded.

We might have to keep the existing link in the developer panel, between links and branches/commits/pull requests for audit purposes, so removing older repositories might not be an option.

Is there any way connecting Jira with Github using something else than DVCS, or other ways of getting around the limit without removing repositories?

Thanks in advance!

[Edit] 
I've looked at the post here, but temporarily raising the limit most likely won't be a long term solution, since this issue have been ongoing for over a week now.

[Andy Heinzer]

Hi Jakob,

Sorry to hear about this problem.  I understand that Jira appears to be unable to sync all the github repo data here due to rate limiting imposed by github.  I also see you referred to Brandon's answer that cited some specifics of the limits here:

You can find more information about GitHub rate limits in Rate limits for GitHub Apps.  Github only allows 5000 requests per hour, regardless of the source.

While reducing the number of repos being synced would like help avoid the rate limit, I can certainly understand why you would not want to go that route here.

I have not found much in the way of alternative tools to do this syncing.  I looked though the marketplace.atlassian.com site, but I didn't exactly find much in regards to Jira Server plugins that seem promising on this front today.

Alternatively, it is possible for you to adjust how often Jira Server is attempting to sync this repo data from github.  Please see the KB How to change the interval or schedule of the DVCS repositories sync.   By default in Jira, the sync happens every hour.  Given you have more than 5000 repos, then it makes sense you would be seeing this rate limit.  Just note that any changes you make directly to the SQL database of Jira typically require you to restart the Jira application after the change has happened to take affect.

You could change the interval here to be every 2 hours (in milliseconds I believe this would be 3600000) or more and I believe that you wouldn't see this rate limit error from Jira again.  Unless of course you have some other service syncing against these repos that might cause you to reach that rate limit sooner.

I hope this helps.

Andy

[Jakob KN]

Hi Andy, 

I had a look around for other options of syncing/linking Github and Jira together, but didn't find any either.

Perhaps i'm not entirely understanding the sync limits.
But, if the entire list of repositories is set to sync every 2nd hour instead of once an hour, wouldn't the hourly limit still be reached due to the number of sync's still being above the limit?

[Andy Heinzer]

That is a good point, you still might encounter this rate limit.  But then again, not necessarily when the interval in Jira is longer than just one hour.  It would depend upon how long it takes Jira to complete that sync operation.   I suggested that approach because in my mind you might be reaching this rate limit before Jira can complete the first sync attempt.  However if that is not the case here, then extending the interval might not be helpful here.  When I read this post the first time, I thought you stated you had 7000 repos, not the 725.  Sorry if I misunderstood your environment here.   But that said, extending the interval might be helpful.   

This rate limit is imposed by github for all kinds of applications that connect to their service.  We don't know if you might have other Jira installations that connect here or some other application that might do the same.  These can all add to the total of the requests that go into that rate limit.  So, it still could be helpful to extend this interval within Jira as a means to troubleshoot this.

I would try that first and see if it helps.   If you've done this change of interval, restarted Jira, and waited a few hours only to see the same problem, then it might be best to look at some other alternatives.

I found an alternative approach that might also be helpful here.  Check out DVCS Github sync failed.   It's a Jira KB for a similar issue.  It documents a dark feature that you could enable in Jira Server that would disable the syncing of pull requests.  This could be a temporary means to at least ensure that the sync of other data such as branches and commits can still be seen.

However if neither of these approaches seem helpful, then I'm not sure what other alternatives would exist.  I would look to ultimately reduce the number of repos that account has to sync.  

Andy