Runners autoscaler 3.11.7 crashes with oauthlib ServerError: (server_error)

[Pavel Ponomaryov]

Runners autoscaler 3.11.7 crashes with oauthlib ServerError: (server_error) — the OAuth token is never cached or retried.

We run the Kubernetes runners autoscaler for a repository runner group, authenticating with an OAuth client (client-credentials grant). The `runner-controller` pod restarts every time the Bitbucket OAuth token endpoint returns a 5xx, because the grant has no retry. Yesterday that happened 15 times in under 7 hours.

Two things combine to make this frequent:

**1. The token is never re-used.** `Auth.token_oauth()` runs a fresh client-credentials grant for every Bitbucket API client the code constructs, and each helper in `services/bitbucket.py` constructs its own. `read_config()` also runs at the top of the poll loop, and its `update_to_uuid` validator resolves the workspace and repository UUIDs over the API on every poll — although those values are static and cannot change.

At a 30s poll interval, that is at least 3 grants per poll, about 8,600 per day, for a token you issue with a 2-hour lifetime. About 12 would do. The response's `expires_in` is already read, then logged at debug level and discarded.

**2. Nothing retries the grant.** The `oauthlib` exception propagates out of `main()` and the container exits.

### Tool version

3.11.7 (the newest tag), image `docker.io/bitbucketpipelines/runners-autoscaler:3.11.7`, running on Kubernetes with one repository group and the `percentageRunnersIdle` strategy.

### Error message

```

Traceback (most recent call last):

File "/home/bitbucket/autoscaler/start.py", line 91, in <module>

main()

File "/home/bitbucket/autoscaler/start.py", line 87, in main

poller.start()

File "/home/bitbucket/autoscaler/start.py", line 32, in start

autoscaler_runners, runner_constants = self.read_config()

File "/home/bitbucket/autoscaler/start.py", line 69, in read_config

runners_data = validators.RunnerData.parse_file(self.config_file_path)

[pydantic validation frames]

File "/home/bitbucket/autoscaler/core/validators.py", line 122, in update_to_uuid

workspace_data, repository_data = BitbucketService.get_bitbucket_workspace_repository_uuids(

File "/home/bitbucket/autoscaler/services/bitbucket.py", line 109, in get_bitbucket_workspace_repository_uuids

repository_api = BitbucketRepository()

File "/home/bitbucket/autoscaler/clients/bitbucket/base.py", line 66, in __init__

self._auth = auth or Auth.token_oauth()

File "/home/bitbucket/autoscaler/clients/bitbucket/base.py", line 30, in token_oauth

token = oauth.fetch_token(Auth.OAUTH_URL, client_id=client_id,

[requests_oauthlib and oauthlib frames]

oauthlib.oauth2.rfc6749.errors.ServerError: (server_error)

```

Kubernetes then reports `CrashLoopBackOff`.

### Steps to reproduce

1. Deploy 3.11.7 on Kubernetes with `BITBUCKET_OAUTH_CLIENT_ID` and `BITBUCKET_OAUTH_CLIENT_SECRET`, and any `percentageRunnersIdle` group.

2. Enable debug logging and count the `Token expires in ...` lines. There are at least three per poll interval.

3. Make `https://bitbucket.org/site/oauth2/access_token` return a single 5xx, or wait for a real one. The container exits.

### Why we cannot work around it by changing the auth method

API-token basic auth is rate-limited against the individual user account, and our controller authenticates as a shared automation identity whose bucket is never quiet. Bitbucket has no workspace-scoped token (see BCLOUD-23004). OAuth gives the controller its own bucket, so we need the OAuth path to be reliable.

Raising `runner_api_polling_interval` reduces the exposure proportionally, but it also slows the ramp, and it does not fix the underlying "one 5xx ends the process" behaviour.

### What we are asking for

1. **Cache the access token** until it is near expiry. `expires_in` is already in the response.

2. **Retry the grant** on transient failures (`ServerError`, `TemporarilyUnavailableError`, connection resets, timeouts) with backoff, while still failing fast on `InvalidClientError` and `UnauthorizedClientError` so real credential problems still surface.

3. **Resolve the workspace and repository UUIDs once**, not on every poll.

4. More broadly, `BaseAPIService.make_http_request` has no retry either, so a 429 on a runner-create POST or a transient connection reset also crashes the controller. A shared retry policy would cover all of these.

This looks like the follow-up to **BCLOUD-23898**, where the team resolved the server-side error rate and said they would "look at ways to improve error handling in the kubes autoscaler in the future". That error handling is what we are missing here.

### Our current workaround

We mount a small launcher that replaces `Auth.token_oauth` with a thread-safe cached version — re-used until 300s before expiry, with bounded retries on transient errors only — and then hands over to the image's own entrypoint. Verified against the 3.11.7 image. Happy to share it if useful, though we would much rather delete it and take a fixed release.