Hello guys, We are running Bitbucket autoscaler version 3.11.7 and we noticed that in dynamic Kubernetes environments with high node churn, the cleaner leaves job and secret definitions when sudenly interrupted during the deletion. Cleaner.delete_runners (autoscaler/cleaner/pct_runner_idle_cleaner.py) deletes the Bitbucket runner first, then its Kubernetes Job and the runner-oauth-credentials- Secret. The cleaner finds runners only by listing them in Bitbucket. If it stops between the two steps (pod restart, rollout, or an error from the Kubernetes API), the next run can't see that runner any more. Its Job and Secret then stay in the cluster for good, including the runner's OAuth credentials. Steps to reproduce * Let the cleaner select an offline or disabled autoscaler runner for deletion. * Kill the cleaner pod right after the Bitbucket DELETE .../pipelines-config/runners/{uuid} call succeeds. * After the pod restarts, the runner is gone from Bitbucket, and runner- and runner-oauth-credentials- still exist in the runner namespace. The fix is to delete the Kubernetes resources before the Bitbucket runner. KubernetesService.delete_job already treats a missing Job or Secret as a warning, so the next run retries safely after an interruption at any point. This: self.kubernetes_service.delete_job(runner_uuid.strip('{}'), self.runner_data.namespace) Should be above this: self.runner_service.delete_bitbucket_runner( workspace=self.runner_data.workspace, runner_uuid=runner_uuid, repository=self.runner_data.repository ) Thanks Stefan

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

runners-autoscaler cleaner orphans runner Jobs and Secrets when interrupted

ll-itops

Hello guys,

We are running Bitbucket autoscaler version 3.11.7 and we noticed that in dynamic Kubernetes environments with high node churn, the cleaner leaves job and secret definitions when sudenly interrupted during the deletion.

Cleaner.delete_runners (autoscaler/cleaner/pct_runner_idle_cleaner.py) deletes the Bitbucket runner first, then its Kubernetes Job and the runner-oauth-credentials-<uuid> Secret. The cleaner finds runners only by listing them in Bitbucket. If it stops between the two steps (pod restart, rollout, or an error from the Kubernetes API), the next run can't see that runner any more. Its Job and Secret then stay in the cluster for good, including the runner's OAuth credentials.

Steps to reproduce

Let the cleaner select an offline or disabled autoscaler runner for deletion.
Kill the cleaner pod right after the Bitbucket DELETE .../pipelines-config/runners/{uuid} call succeeds.
After the pod restarts, the runner is gone from Bitbucket, and runner-<uuid> and runner-oauth-credentials-<uuid> still exist in the runner namespace.

The fix is to delete the Kubernetes resources before the Bitbucket runner. KubernetesService.delete_job already treats a missing Job or Secret as a warning, so the next run retries safely after an interruption at any point.

This:

self.kubernetes_service.delete_job(runner_uuid.strip('{}'), self.runner_data.namespace)

Should be above this:

self.runner_service.delete_bitbucket_runner(
    workspace=self.runner_data.workspace,
    runner_uuid=runner_uuid,
    repository=self.runner_data.repository
)

Thanks

Stefan

Accepted answers

Mihai_LeanZero

Hi @ll-itops,

The repo has issues turned off, and its README sends bug reports and feature requests to Community instead. So this thread already counts.
https://bitbucket.org/bitbucketpipelines/runners-autoscaler/src/master/README.md

Closest public ticket is BCLOUD-23300, about orphaned runner resources on Kubernetes but not the deletion order, and it's worth a vote.
https://jira.atlassian.com/browse/BCLOUD-23300

As far as I can tell, BCLOUD bugs get filed by Atlassian staff, so on a paid plan a support ticket with the repro steps is the route to a public bug. Master and 3.11.7 still remove the Bitbucket runner first.

For the leftovers, the Secret carries a runner_uuid label and the Job picks up the same one from its pod template. Match those against the runners Bitbucket still lists for every workspace and repo the autoscaler groups use, otherwise live runners get flagged. The label has no braces. I haven't tested that sweep, so start with one runner.

All comments

shivain gupta

You're right about the cause, and your proposed order is the right fix. The cleaner lists runners from Bitbucket, so once the Bitbucket runner is deleted, the next run has no way to find the leftover Job and Secret — deleting the Kubernetes resources first makes an interruption at any point safe to retry, since a missing Job or Secret is already treated as a warning. Two things worth doing alongside the reorder: sweep the runner namespace for orphaned runner Jobs and Secrets with no matching Bitbucket runner and delete them, since those Secrets hold runner OAuth credentials and shouldn't sit around; and if you haven't already, raise this as an issue or PR against the autoscaler repo itself, because a community thread can't change the deletion order in the code.

ll-itops

@shivain gupta Thanks for prompt response. We tried to open a PR into the repo but we don't seem to have permissions to do that. We cannot even fork the repo, bitbucketpipelines/runners-autoscaler.

We also tried to create an issue but I don't see relevant project for this. We found issues in Bitbucket Cloud project but it does not seem to be available for us.

Can you advise here please?

Thanks

Stefan

Mihai_LeanZero

Hi @ll-itops,

Closest public ticket is BCLOUD-23300, about orphaned runner resources on Kubernetes but not the deletion order, and it's worth a vote.
https://jira.atlassian.com/browse/BCLOUD-23300

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

kubernetes

bitbucket-runner

runners-autoscaler