Problems with Jira's DVCS Github repository sync coming unstuck and not finding new changes

Question

We've had our local and self hosted GitHub Enterprise instance connected to our Jira Software (Server) instance for a few years now and in recent months we've found countless scenarios where the regular sync has become unstuck.

What I mean by this is that many commits, PR's or branches with the correctly formatted Jira KEY-1234 reference in all caps are being missed.

Worse that this seems to be in repositories which were proven to be working just days, weeks, or months ago.

Whenever we've inspected this, the DVCS administration page has implied that there's been no activity in X days or weeks. We then force a sync on that repo alone and after a few minutes it finds any relevant commits, branches or PR's. This isn't sustainable and is impacting more repos every day.

The DVCS service is scheduled for a Sunday night so as to minimise impact to our 2000 users.

We do have many hundreds of orgs, thousands of repositories and millions of commits etc - so my only guess is that the synchronisation process timesout due to how much it must trawl through t ofind new commits etc.

Has anyone else faced this same issue in their enterprise? And if so what changes have you put in place to resolve this issue

Side note: post commit hooks connections can not be setup on these repos, so we rely on Jira's sync itself.

Mike Rocheleau · Answer

I fail at updating in a timely fashion, but the error in my case (error 422) turned out to be a Github bug in the version we were on at the time.  They had some higher restrictions in place on testing validity of the call from JIRA or somesuch, as part of some other thing but they never rolled out the other thing but accidentally left these restrictions in place (they = github).

Anyway we upgraded Github to the next release and that fixed it.

Mike Rocheleau · Answer

Just to chime in, I've got this exact scenario happening.  The issue is the Hooks in the Github side for the repos are vanishing, or at least if I try to re-create the connection for a failing Org/repo, it throws a "could not create webhook" error in JIRA, saying error 422.

On the Github side, in the unicorn.log file you'll find the Github side of that error, at least in our case, we're seeing an error returned of "errors='"id" is not a permitted key.'"

Atlassian Support is currently chasing this down as it Github support, I'll update this is anything gets figured out.

The weird thing is everything worked for years, we presume, no one complained at least.  Only in the past few months has this come up as a problem for us, but maybe it's only been noticed in the past few months, but has been failing for a while.

Not all repos/orgs are failing though, some have hooks still and they work, some seem to be working even without hooks showing up in the repo webhook settings page.  I don't understand how those are working at all...