Possibility of Spawning Dynamic Runners?

In order to create and register a runner, we have to use the bitbucket.org web interface to create an individual runner, then we can copy down the relevant docker commands for creating a single runner.

I'm curious about whether it is possible, or perhaps someday will be possible, to dynamically spawn a runner. For example, it would be nice if I could manage a cluster of runners via Kubernetes or Docker Swarm given some kind of master credentials that allowed for creating runners on the fly.

13 comments

Comment

Log in or Sign up to comment
Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 8, 2021

@Chad Gilbert Sorry for the delayed reply. Yes, autoscaling of self-hosted runners on a K8s cluster is on our long-term roadmap. In the short term, we will be releasing the APIs so that you can build a custom solution to automate the creation of the runners. We are currently aiming at releasing the APIs in the next quarter. 

I would like to dig a bit deeper into the problem you are trying to solve. When would you auto-scale runners?

Thanks,

Justin Thomas

Like Artsiom Zhurbila likes this
Chad Gilbert September 9, 2021

I think my most pressing reason for asking this question revolves around the tedium of registering and upgrading a cluster of runners: We have no option at present other than managing each runner by hand in the Bitbucket interface and running appropriate Docker commands to upgrade (or remove and recreate) each container, often meaning logging into multiple hosts and running the same kinds of commands over and over. Right now, it's just a lot of legwork any time an upgrade is needed, and there have been several times where a breaking change in Runners forced us to upgrade our cluster before builds could be resumed.

In my particular case, we've just got some commodity hardware sitting in-house acting as runners, and that suffices for our case right now, so auto-scaling in and of itself isn't that high a priority for me, but it seems to coincide with the automatic creation of runners.

However, your comment about releasing API endpoints for runner automation sounds very promising, and given that, I would probably try to cook up an automation solution that would fit our needs. Thanks for the update!

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 13, 2021

Thank @Chad Gilbert for explaining the use case. We are aiming to make the Runners API public in the next quarter. Auto-updating of runners is something that we have been investigating, would an auto-updating running fit your use case?

Like Chad Gilbert likes this
Chad Gilbert September 13, 2021

An auto-updating runner would definitely ease some of the regular burden! I look forward to the API updates in the next quarter as well. Thanks!

Like Justin Thomas likes this
Artsiom Zhurbila December 9, 2021

@Justin Thomas any ETA to release Runners API? It's super useful for us to have ability to register Workspace runner using k8s job script in the beginning of pipeline and release this job at the end of pipeline

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
December 13, 2021

@Artsiom Zhurbila Yes, we understand the importance of public APIs and it's our second priority after Windows. The team is currently working on Windows Runners and we are planning to make the Runners API public early next year after Windows Runners is in open beta. So hopefully in Q1 CY22 the Runners API should be public.

Siva Abbannagari June 27, 2022

Any update on AutoRunner API?

Shabeeb Khalid September 6, 2022

And what is the status of the above said API? sad-boy-depressed-boy-looking-lonely-free-vector.jpeg

Like Zakeeruddin M likes this
Zakeeruddin M May 12, 2023

@Justin Thomas Any updates on the API that was promised late 2021 ??

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
May 14, 2023
Florian Besser June 22, 2023

@Justin Thomasthe URL you posted seems to no longer be available.

 

We're facing the issue where we during the day have a higher load and would want to scale our custom runners (they run in AWS EC2). Right now we must manually add runners and then start /stop them but we can't get any automated triggers i.e. "all current runners are busy, add another runner"

 

GitHub seems to have something like this: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners, and we're also looking at CircleCI if BitBucket cannot satisfy such a requirement.

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
June 25, 2023

@Florian Besser Oops, the URL points to a private community. @Liam Nunns should be able to add you to it

Like # people like this
TAGS
AUG Leaders

Atlassian Community Events