API rate limiting and Jenkins SCM polling

[#1]

Hello All,

Hope everyone is well out there. This article will focus on the impact of Source Code Management (SCM) polling in Jenkins and Bitbucket Cloud Rate Limits.

What is SCM polling in Jenkins?

Bitbucket Branch Source Plugin

User Guide

The plugin is designed to poll Bitbucket cloud repositories for changes based on the interval as specified by a user. The changes can include: new commits, branches, pull requests. This is a useful feature that many Bitbucket Cloud Customers use. Especially the Bitbucket Branch Source Plugin (installed approximately 65555 times)

What is API Rate Limiting?

In general, rate limiting is used to protect overuse of API, by limiting the amount of calls an authenticated user can make the the application over API. It also helps the provider prevent flooding and request queues that will cause delays caused by “noisy neighbours“ and to identify and deflect DDoS attack. Another application of rate limit is to make the API scalable, a sudden increase in the usage of a new/popular API can cause a traffic spike causing severe lags.

API Rate Limiting in Bitbucket Cloud

For current Bitbucket Rate Limits, please check our support documentation;

Bitbucket Cloud API Rate Limiting

Now that we have some of the key information out of the way, let's get to the situation at hand.

Over a period of time, we have noticed our users who integrate their accounts with Jenkins set up SCM polling from the Jenkins side. When the amount of API calls reaches the limit due to polling per endpoint per user, Bitbucket will not accept any more calls and the response will be 429 to indicate that the user reached its limit per current hour. This causes frustrating issues such as pages not loading on time, having to refresh multiple times to see even the tiniest of data.

Since there is no explicit information on the UI mentioning API rate limit in place, this can lead to a frustrating day indeed. You might be greeted with multiple “something went wrong” error akin to this;

What we can do here is to open up our Browser’s network tab, developer console and look for a “429” error which would indicate Rate Limiting.

So this can help you identify what is happening, but that is not quiet enough, let’s see how we can work around while using SCM polling that is interacting with Bitbucket.

Adjust the Polling

If at all possible, the focus can be on increasing the time between requests. Adjust the polling in a way that it does not overwhelm the system with requests and triggers rate limiting from Bitbucket platform. If this is not possible, and if you need to do the polling at that rate, then we can look at the second option;

Set Up a dedicated CI/CD account in Bitbucket Cloud

Here, you would have to sign up for a new Bitbucket Cloud account, you can perform your Jenkins Integration and SCM polling to this account, this will make sure that your main individual account that you use for UI navigation and anything else in Bitbucket will not be rate limited.

One thing to remember here is that this new user will be counted towards your workspace plan. As I write this, this extra user a topic of discussion internally and any updates on this will be shared with all our customers.

Note: I hope this article provided you with an insight on why rate limiting happens on Jenkins SCM polling and the ways to work with/around it.

Finally, if you have any other issues, API or otherwise, you can always find us at Bitbucket Cloud Support

Cheers,

Stay Safe & Have a wonderful day!
Regards
Manu Abraham
Bitbucket Cloud