How to do maintenance on local runners

Question

We are running local runners for Bitbucket pipelines. We have multiple, and I need to make them low effort to operate them. That means that auto updates have been enabled (including reboot) and I also want to auto update the agent itself. As the runners are Linux based and are run using docker, I plan to setup watchtower.

How do I make sure the update does not interrupt a (potential) running build?

Theodora Boudale · Answer

Hi Hans,

I'm afraid that there is no API endpoint that returns the state of the runner and whether it's in use. The app password permissions for runners are there for users who use the Runners Autoscaler on Kubernetes:

* https://support.atlassian.com/bitbucket-cloud/docs/autoscaler-for-runners-on-kubernetes/

I understand the following:

That will take some time (at least seconds) by the time I have checked, it could very well be that a build has started.

However, that could still be an issue with an API that returns the runner status, as a new job may be assigned to the runner in the time between checking the runner status and stopping the runner.

We have feature requests both for APIs that return the runner status and also for an auto-update option for the runners:

* https://jira.atlassian.com/browse/BCLOUD-21309
* https://jira.atlassian.com/browse/BCLOUD-22032

In the meantime, if your builds run 24/7 and there isn't a day and time that the runners are certainly idle, checking the pipelines is the only available method to find out if there are running builds.

Kind regards,
Theodora

Hans de Groot · Answer

So I am unable to check the actual state of the runner?

You want me to check all my repositories for a potential build? That will take some time (at least seconds) by the time I have checked, it could very well be that a build has started.

But then why is there a permission related to runners: https://support.atlassian.com/bitbucket-cloud/docs/app-password-permissions/#Runners

But there seems to be no endpoint for runners?

Theodora Boudale · Answer

Hi Hans,

The first step before updating a runner is to stop the runner and doing so will also interrupt any step running on this runner.

You could use our API to get all repos in a workspace and all pipelines in a certain repo that have a status of PENDING, BUILDING, or IN_PROGRESS:

* https://developer.atlassian.com/cloud/bitbucket/rest/api-group-repositories/#api-repositories-workspace-get
* https://developer.atlassian.com/cloud/bitbucket/rest/api-group-pipelines/#api-repositories-workspace-repo-slug-pipelines-get

One of my colleagues posted in another community question an example API call with curl that returns in progress pipelines for a certain repo:

* https://community.atlassian.com/t5/Bitbucket-questions/Re-Re-Possible-value-for-status-filtering-in-API-pipelines/qaq-p/2071030/comment-id/83461#M83461

If this call returns running pipelines, you can delay the update and retry later when there are no running pipelines.

Please keep in mind that the results do not indicate if a certain step runs on a self-hosted runner or on Atlassian's infrastructure. If you want this info and you also want to know which runner is used, you could get the steps of the running pipeline, then get the log file of the running step and parse it to see if that step runs on a runner and if so, what is the runner's UUID. These are the API endpoints for retrieving the steps of a pipeline and the log of a certain step:

* https://developer.atlassian.com/cloud/bitbucket/rest/api-group-pipelines/#api-repositories-workspace-repo-slug-pipelines-pipeline-uuid-steps-get
* https://developer.atlassian.com/cloud/bitbucket/rest/api-group-pipelines/#api-repositories-workspace-repo-slug-pipelines-pipeline-uuid-steps-step-uuid-log-get

Alternatively, if you know that all your self-hosted pipelines are finished by a certain time in the day and they only start in the morning of the next working day, you could schedule the update during that timeframe of inactivity or during the weekend (if you have no pipelines running in the weekend). If you use scheduled pipelines, you would also need to take into account the time that you schedule them and the time they usually need to finish, so that you schedule the update accordingly.

Kind regards,
Theodora