Jira DVCS Connector exceeds GitHub API request quota and duplicates commits/issues

MrMage_

Ever since I upgraded from Jira 8.14 to 8.18, I am seeing the Jira DVCS Connector cause an excessive amount of requests to the GitHub API, as well as vastly overestimate the number of commits and issues in my repositories.

GitHub support has told me that Jira alone has sent more than 200,000 requests to the GitHub API in a single week, for a fairly small organization (we are 3 people in total).

In addition, the Jira DVCS Connector reports almost 30k commits and 10k issues for one of my repositories, while GitHub reports 7k commits and 250 issues total, so clearly something is off here:

I am using Jira Software Server. I have since upgraded from 8.18 to 8.18.1 (the release notes for 8.18.1 list several fixed issues with the DVCS Connector), but my issue is still present.

I have already removed my GitHub connection in the DVCS Connector altogether and set it up with a new token, to no avail.

Kirill Ivanov_f1f64m

We have the same issue after upgrade from 8.16 to 8.18.1. But we used Self-managed GItLab instance. Not GitHub.  So we have no problem with api rate limiting. In the logs of the GitLab server, you can see that at the begining of the synchronization process, Jira makes the following HTTP GET requests:

Started GET "/api/v4/groups/apm/projects?pagination=keyset&order_by=id&last_activity_after=2021-07-29T16:43:31.283Z&membership=true&simple=true&include_subgroups=true&per_page=100" for 10.20.160.
2 at 2021-07-29 17:49:28 +0000
Started GET "/api/v4/groups/mpos/projects?pagination=keyset&order_by=id&last_activity_after=2021-07-29T16:43:31.695Z&membership=true&simple=true&include_subgroups=true&per_page=100" for 10.20.160
.2 at 2021-07-29 17:49:29 +0000
Started GET "/api/v4/groups/oi/projects?pagination=keyset&order_by=id&last_activity_after=2021-07-29T16:43:31.966Z&membership=true&simple=true&include_subgroups=true&per_page=100" for 10.20.160.2
at 2021-07-29 17:49:29 +0000
Started GET "/api/v4/groups/projectoffice/projects?pagination=keyset&order_by=id&last_activity_after=2021-07-29T16:43:32.233Z&membership=true&simple=true&include_subgroups=true&per_page=100" for
10.20.160.2 at 2021-07-29 17:49:29 +0000
Started GET "/api/v4/groups/ps/projects?pagination=keyset&order_by=id&last_activity_after=2021-07-29T16:43:32.525Z&membership=true&simple=true&include_subgroups=true&per_page=100" for 10.20.160.2
at 2021-07-29 17:49:30 +0000
Started GET "/api/v4/groups/ps/projects?pagination=keyset&order_by=id&last_activity_after=1970-01-01T00:00:00Z&membership=true&simple=true&include_subgroups=true&per_page=100" for 10.20.160.2 at
2021-07-29 17:49:30 +0000

After enabling additional logging for the Jira Dvcs plugin (according to the article Jira debug logging reference ), we see the following messages in the logs:

2021-07-29 17:56:09,272+0000 DVCSConnector.MessageExecutor:thread-3 DEBUG ServiceRunner [c.a.j.p.d.spi.gitlab.GitLabCommunicator] Fetched [38] files for commit [5e7cfdacc3b0003521fc1876afff74a1158a6fcc] for respository [paymentgate-templates] in organization [ps]
2021-07-29 17:56:09,272+0000 DVCSConnector.MessageExecutor:thread-3 DEBUG ServiceRunner [c.a.j.p.dvcs.sync.MessageConsumerSupport] Marking changeset node = 5e7cfdacc3b0003521fc1876afff74a1158a6fcc to be processed by smart commits with value = true
2021-07-29 17:56:09,288+0000 DVCSConnector.MessageExecutor:thread-3 DEBUG ServiceRunner [c.a.j.p.d.service.message.MessageConsumer] Could not acquire token for consumer: 'com.atlassian.jira.plugins.dvcs.sync.GitLabSynchronizeChangesetMessageConsumer@324c7f9' with id 'com.atlassian.jira.plugins.dvcs.spi.gitlab.message.GitLabSynchronizeChangesetMessage' message '[id=293922,address=com.atlassian.jira.plugins.dvcs.service.MessageAddressServiceImpl$MessageAddressLoader$1@54423897,priority=10,retriesCount=0,payloadType=class com.atlassian.jira.plugins.dvcs.spi.gitlab.message.GitLabSynchronizeChangesetMessage,payload={"repository":355,"softSync":true,"webHookSync":false,"version":1,"node":"5e7cfdacc3b0003521fc1876afff74a1158a6fcc","branch":"ASSIST-3860","refreshAfterSynchronizedAt":1627580976385},tags={synchronization-repository-355,audit-id-1043}]'

 

We have verified that everything is in order with the Oauth2 token. And we are not completely sure that this message affects the problem that has arisen. If you need any additional information, we are ready to provide it.