This article is a result of extensive research and practical experience in setting up and running Runner auto scaler on an on-premise Kubernetes cluster setup at a mid-stage startup company. This is meant as a quick start guide for engineers to get started with Runner auto scaler quickly to make their runners easy to extend and operate at scale.
Prerequisites
- Access to Bitbucket Cloud Premium version (You need a Premium account for Runner autoscaler specifically) and necessary credentials for the Runner Autoscaler. Please refer to the setup guide here: Autoscaler for Runners on Kubernetes: Bitbucket Cloud Docs.
- Note: I suggest the use of OAuth Tokens as they are easy to create, scale, and maintain. OAuth credentials must be Base64-encoded for Kubernetes secrets.
- At least two VMs with network access to each other:
- 1 Master Node
- 1 Worker Node (you can have multiple worker nodes, but ensure they have network access to the master).
Step 1: Prepare the Master Node
Recommended: Go through the official MicroK8s documentation and understand how MicroK8s works before proceeding.
1. Update and Install Necessary Tools
sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt install -y apt-transport-https curl gnupg lsb-release
2. Install MicroK8s
sudo snap install microk8s --classic
sudo microk8s status --wait-ready
3. Configure Network Settings (Optional)
sudo vi /etc/netplan/00-installer-config.yaml
sudo netplan apply
4. Set the Hostname
hostnamectl set-hostname kubemaster # or any other name you prefer
5. Add Worker Nodes
Generate a join command on the master:
sudo microk8s add-node
Follow the on-screen instructions to join worker nodes using the generated token.
6. Install Helm
You need Helm to install the Runner Autoscaler as it is packaged as a Helm chart.
sudo snap install helm --classic
Step 2: Prepare the Worker Nodes
1. Update and Install MicroK8s
sudo apt-get update -y
sudo apt install -y vim
sudo snap install microk8s --classic
sudo microk8s status --wait-ready
2. Join the Worker Node to the Master Node
sudo microk8s join <MASTER_NODE_IP>:<PORT>/<TOKEN> --worker
Step 3: Clone and Configure the Runner Autoscaler
1. Clone the Runner Autoscaler Repository
git clone git@bitbucket.org:bitbucketpipelines/runners-autoscaler.git
cd runners-autoscaler/kustomize
git checkout 3.7.0 # Make sure to check out the latest available version
2. Configure Runner Autoscaler
Edit the runners_config.yaml and kustomization.yaml files to include your Bitbucket OAuth credentials.
sudo vi values/runners_config.yaml
sudo vi values/kustomization.yaml
Example runners_config.yaml
constants:
default_sleep_time_runner_setup: 10 # value in seconds
default_sleep_time_runner_delete: 5 # value in seconds
runner_api_polling_interval: 600 # value in seconds
runner_cool_down_period: 300 # value in seconds
groups:
- name: "Linux Docker Runners"
workspace: "YOURWORKSPACENAME" # workspace name
labels:
- "self.hosted"
- "linux"
- "runner.docker"
namespace: "default"
strategy: "percentageRunnersIdle"
parameters:
min: 4
max: 8
scale_up_threshold: 0.5
scale_down_threshold: 0.2
scale_up_multiplier: 1.5
scale_down_multiplier: 0.5
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "4Gi"
cpu: "2000m"Example kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
configMapGenerator:
- name: runners-autoscaler-config
files:
- runners_config.yaml
options:
disableNameSuffixHash: true
namespace: bitbucket-runner-control-plane
commonLabels:
app.kubernetes.io/part-of: runners-autoscaler
images:
- name: bitbucketpipelines/runners-autoscaler
newTag: 3.7.0 # Ensure this matches the version you checked out earlier.
patches:
- target:
version: v1
kind: Secret
name: runner-bitbucket-credentials
patch: |-
### Option 1 ###
- op: add
path: /data/bitbucketOauthClientId
value: "ENTER_BITBUCKET_OAUTH_CLIENT_ID_HERE_WITHIN_QUOTES"
- op: add
path: /data/bitbucketOauthClientSecret
value: "ENTER_BITBUCKET_OAUTH_CLIENT_SECRET_HERE_WITHIN_QUOTES"
### Option 2 ###
# - op: add
# path: /data/bitbucketUsername
# value: ""
# - op: add
# path: /data/bitbucketAppPassword
# value: ""
- target:
version: v1
kind: Deployment
labelSelector: "inject=runners-autoscaler-envs"
patch: |-
### Option 1 ###
- op: replace
path: /spec/template/spec/containers/0/env
value:
- name: BITBUCKET_OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
key: bitbucketOauthClientId
name: runner-bitbucket-credentials
- name: BITBUCKET_OAUTH_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: bitbucketOauthClientSecret
name: runner-bitbucket-credentials
### Option 2 ###
# - op: replace
# path: /spec/template/spec/containers/0/env
# value:
# - name: BITBUCKET_USERNAME
# valueFrom:
# secretKeyRef:
# key: bitbucketUsername
# name: runner-bitbucket-credentials
# - name: BITBUCKET_APP_PASSWORD
# valueFrom:
# secretKeyRef:
# key: bitbucketAppPassword
# name: runner-bitbucket-credentialsStep 4: Validate the Setup
1. Check Node Status (On Workers)
sudo microk8s kubectl get nodes
2. Check Pod Status
sudo microk8s kubectl get pods -n bitbucket-runner-control-plane --field-selector=status.phase=Running
3. Inspect Runner Logs
sudo microk8s kubectl logs -f runner-controller-<pod-name> -n bitbucket-runner-control-plane
Additional Tips
- Scaling Runners: The autoscaler should handle scaling runners dynamically based on job demand. Ensure your runner configurations are correct in
runners_config.yaml. - Resource Monitoring: Use the following command to monitor resource usage and adjust configurations if necessary:
sudo microk8s kubectl top nodes
Troubleshooting
- Failed Pod Scheduling: If runners fail to schedule, check node taints and pod events.
- Logs Not Displaying: Ensure
--max-log-requests is not limiting the logs excessively.
Additional Improvements
Use MicroCeph or just Ceph to manage storage that backs the MicroK8s nodes for better flexibility and scalability. Refer to the MicroCeph Multi-Node Install Guide.
