Is there a way to programatically create a new runner? (Not using the Bitbucket web interface)

[kalib]

Hi, is it possible to create a runner in any programmatic or automated way? I know we need to go to the web interface and go to the settings and create a runner, copy that docker command and start that command in a host. I was wondering if there's a way to do that automatically, maybe via an API call or something. The idea is that I wouldn't need to have dozens of runners up and running 24/7 and to be able to spin up runners at specific times or even as a step of some pipeline for example.

But since bitbucket pipelines requires that specific ID for each runner that is generated only during the creation of the runner, I was wondering if there would be a way to do that via command line somehow, so I can automate the creation of the host and starting the docker container there automatically.

[jawad_abbasi]

Hi @Mark C what's an update about that ticket to build some api to create bitbucket runner automatically ?

[Mark C]

Hi @kalib

Thank you for reaching out to the community.

Unfortunately, it's not possible to create Pipelines Runners programmatically using REST API at the moment.
We do have an existing feature request for it that can be located through this link. - https://jira.atlassian.com/browse/BCLOUD-21708
You can upvote and watch it for now so that you'll be notified of any updates from our team when the feature becomes available on Bitbucket Cloud.
Please do note that we don't have an exact ETA for the feature request as all new features will be implemented according to our policy here.

Let me know if you have further questions that I can help with.

Regards,
Mark C

[Einar Coutin]

Thanks for your policy document. So if the permissions for managing runners are already implemented in the API:
https://developer.atlassian.com/cloud/bitbucket/rest/intro#runner

And there are TWO tickets requesting it:
[BCLOUD-21309] Have public API endpoints for pipelines runners - Create and track feature requests for Atlassian products.

[BCLOUD-21708] API for creating a new Runner - Create and track feature requests for Atlassian products.

Were there already plans to do this before it was requested?

The use cases are pretty obvious from an auto-scaling perspective, I mean since it has already been mentioned in these forums that there's a requirement of setting docker in docker for the bitbucket runners  combined with the fact that they're configured via an UUID at creation time rather than at registration time makes it all more cumbersome to be launching docker containers/runners with a FIXED UUID environment variable that comes from the api when the runner is registered rather than registering the runner with a UUID given by your cloud endpoint after a slot for it has been created in the workspace/repository when it calls it.

With that said, we understand that Bitbucket Cloud Runners(hosted by you) Autoscale automatically. But two limitations are present:
1.-There's no On Prem Connectivity outside of a DMZ.
2.-The limitation of only having up to 8GB of RAM limits runner usage.

Combine those with the fact of how the Self-Hosted  runners flow works (where the registration token is created and given to the user AFTER the fact) and there's not much we can do to auto-scale self-hosted runners, even if we created an API outside of Bitbuckets: for example AWS Lambda and Cloudwatch, it wouldn't work because runners can't use the aws-logger docker driver to send data to Cloudwatch, which makes sense since the json-logger driver is required in order to collect and send data to Bitbucket cloud.

[Mark C]

Hi @Einar Coutin

I totally understand that.

Indeed, those BCLOUD tickets you mentioned are identical.
Hence, I went ahead and marked the latter as a duplicate.

Regarding Bitbucket Cloud Pipelines Runners auto-scaling, we have an existing feature request for it here where you can vote/watch as well: https://jira.atlassian.com/browse/BCLOUD-21343

Alternatively, if you're using Kubernetes, we've recently shipped a Kubernetes Autoscaler for Runners that is currently in Early Access Preview (EAP).
You can join the group here for more details on how to use it: https://community.atlassian.com/t5/Bitbucket-Pipelines-Runner/gh-p/bitbucket-runner-autoscaler-4k8s

Regards,
Mark C

[Einar Coutin]

Thanks for your reply Mark!

I've requested to join the group. Regardless of autoscaling environment (whether Kubernetes, EC2, etc) the question remains as to how are the runners registering against the Bitbucket cloud database in order for pipelines to identify them and use them.

You can't put the cart in front of the ox so to speak. And right now there's no way to get a runner's UUID without registering MANUALLY on the API first. That's the challenge, not creating copies of the agent.

Do you have any insights into that part of the solution?

Thank You!

[Mark C]

Hi @Einar Coutin

I'd like to kindly address the concern about the confusion from the API authentication page. - https://developer.atlassian.com/cloud/bitbucket/rest/intro#runner

The API authentication scopes for Runners are meant for internal use only.
Meaning, those scopes are used for calling internal APIs related to Runners.
However, there are still no public API endpoints for Runners.

I've raised this to the team through this link because it is confusing for users to see the Runners scopes while there are no public APIs for it yet. - https://jira.atlassian.com/browse/BCLOUD-22192

For the feature request about Runners API endpoints, our product team confirms that it's not part of our short-term roadmap at the moment.
Although it may change in the future, for now, the best I can suggest is for you to continue to watch over the feature request BCLOUD-21309 for any updates from our team.

Regards,
Mark C

[Marc]

It's an obvious feature (and an obviously fragmented roadmap) to be missing a CRUD API for a first-class released feature (runners), for two years.

The duration it remains unaddressed, and the problems inherent in how runners are designed makes it patently clear -- our organization should seek alternative solutions providers or products.

[yoanis]

No worries or hurry ... they'll get it done by 2028 :)

[Andres Mägi]

Warm suggestion, switch to Gitlab, seems to be lightyears ahead in pipeline automation space.