runners-autoscaler cleaner

[Deleted user]

Hello!

I'm using runners-autoscaler for k8s for couple of weeks and it works great. I also want to have separate runner for macos.

How to prevent runners-autoscaler cleaner from removing that runner when it is not active? It there a way of excluding some runners from being cleaned? :)

[Igor Stoyanov · Accepted solution]

@[deleted]  hi. Thanks for your question.

 The first option for your case is if you do not need cleaner at all, you could not set up cleaner-deployment.yaml.

The second option if you still need a cleaner you could increase the value of 

`runner_cool_down_period` in configMap so cleaner will not delete runners that currently created less than this period before.
Also after this feature will be released then the cleaner will not delete runners that had their state updated (instead of created) created less than period before.

Is these options covers your case? Any feedback appreciated.

Regards, Igor

[Deleted user]

@Igor Stoyanov thanks for the answer.

Not exactly what I need. I have some runners which are being created by runners-autoscaler. It automatically adds and removes runners depending on load. I'm happy with that.

But, I want to add completely separate runner on standalone EC2 instance for macOS. But when that runner is not active cleaner removes it.

So my question is it possible to prevent cleaner from removing one specific runner(for example by tag) or tell cleaner to only clean runners which were created by runners-autoscaler only? 

[Igor Stoyanov]

Current logic is the next:
Cleaner looks at you configMap, and if you have the next group on workspace:

- name: "Runner group 1"
  workspace: "test"
  labels:
    - "demo2"
  namespace: "my-namespace"
  strategy: "percentageRunnersIdle"
  strategy_data:
  parameters:
    ...

then cleaner will fetch all the runners from Bitbucket API and will delete them according to logic (not online state and created long time ago).

One of the possible solution to prevent all fetched runners from deletion is to filter runners with some condition (you suggested by tag, it's label in our logic implementation i think). This is what we currently thinking about and gathering interests.

There is pros and cons for this solution.

The cons is if you replace the group above with another group (in example for workplace only but with different labels) than some offline runners created by our tool will never be deleted automatically by cleaner.

"or tell cleaner to only clean runners which were created by runners-autoscaler only"  -- this is a good idea, we could add some default label i.e `autoscaler-created` to runners and then fetch runners to delete filtered by this label.
We will investigate this and notify you if this feature will be released.


Thank you for your suggestion.
Regards, Igor

[Deleted user]

@Igor Stoyanov thank you for the answer.

"this is a good idea, we could add some default label i.e `autoscaler-created` to runners and then fetch runners to delete filtered by this label."

Exactly, where I can vote for this feature? :) 

[Igor Stoyanov]

@[deleted] . I already added this task to our Jira, so no need to vote for it.

But generally the process is the next.

Here is the list of tickets, where you could create and track your feature request.

For this feature we had internal request, but we decide to wait for the customers feedback, so you are the first one who encourage us to implement this.

Regards, Igor

[Deleted user]

Thank you! Eagerly waiting for it to be released!

[Igor Stoyanov]

 @Serhii Kaidalov  hi. We released a new version of the autoscaler tool 3.2.1, that requires for cleaner to check the next to assume should runners be deleted or not:

• runners should not have status "ONLINE",
• runners should have autoscaler.created label (this label was automatically added when runner created by runners autoscaler tool),
• runners should have their state updated more than some period of time ago. You can tune it with "runner_cool_down_period" variable from ConfigMap runner-config.

The `autoscaler-created` label will be added automatically when autoscaler tool will create runners.

Regards, Igor

[Deleted user]

Awesome, thank you!