Setting Up Bitbucket Cloud Runner Autoscaler using MicroK8s

[#1]

This article is a result of extensive research and practical experience in setting up and running Runner auto scaler on an on-premise Kubernetes cluster setup at a mid-stage startup company. This is meant as a quick start guide for engineers to get started with Runner auto scaler quickly to make their runners easy to extend and operate at scale.

Prerequisites

• Access to Bitbucket Cloud Premium version (You need a Premium account for Runner autoscaler specifically) and necessary credentials for the Runner Autoscaler. Please refer to the setup guide here: Autoscaler for Runners on Kubernetes: Bitbucket Cloud Docs.
• Note: I suggest the use of OAuth Tokens as they are easy to create, scale, and maintain. OAuth credentials must be Base64-encoded for Kubernetes secrets.
• At least two VMs with network access to each other:
  • 1 Master Node
  • 1 Worker Node (you can have multiple worker nodes, but ensure they have network access to the master).

Step 1: Prepare the Master Node

Recommended: Go through the official MicroK8s documentation and understand how MicroK8s works before proceeding.

1. Update and Install Necessary Tools

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt install -y apt-transport-https curl gnupg lsb-release

2. Install MicroK8s

sudo snap install microk8s --classic
sudo microk8s status --wait-ready

3. Configure Network Settings (Optional)

sudo vi /etc/netplan/00-installer-config.yaml
sudo netplan apply

4. Set the Hostname

hostnamectl set-hostname kubemaster # or any other name you prefer

5. Add Worker Nodes

Generate a join command on the master:

sudo microk8s add-node

Follow the on-screen instructions to join worker nodes using the generated token.

6. Install Helm

You need Helm to install the Runner Autoscaler as it is packaged as a Helm chart.

sudo snap install helm --classic

Step 2: Prepare the Worker Nodes

1. Update and Install MicroK8s

sudo apt-get update -y
sudo apt install -y vim
sudo snap install microk8s --classic
sudo microk8s status --wait-ready

2. Join the Worker Node to the Master Node

sudo microk8s join <MASTER_NODE_IP>:<PORT>/<TOKEN> --worker

Step 3: Clone and Configure the Runner Autoscaler

1. Clone the Runner Autoscaler Repository

git clone git@bitbucket.org:bitbucketpipelines/runners-autoscaler.git
cd runners-autoscaler/kustomize
git checkout 3.7.0 # Make sure to check out the latest available version

2. Configure Runner Autoscaler

Edit the runners_config.yaml and kustomization.yaml files to include your Bitbucket OAuth credentials.

sudo vi values/runners_config.yaml
sudo vi values/kustomization.yaml

Example runners_config.yaml

constants:
  default_sleep_time_runner_setup: 10 # value in seconds
  default_sleep_time_runner_delete: 5 # value in seconds
  runner_api_polling_interval: 600 # value in seconds
  runner_cool_down_period: 300 # value in seconds

groups:
  - name: "Linux Docker Runners"
    workspace: "YOURWORKSPACENAME" # workspace name
    labels:
      - "self.hosted"
      - "linux"
      - "runner.docker"
    namespace: "default"
    strategy: "percentageRunnersIdle"

    parameters:
      min: 4
      max: 8
      scale_up_threshold: 0.5
      scale_down_threshold: 0.2
      scale_up_multiplier: 1.5
      scale_down_multiplier: 0.5

    resources:
      requests:
        memory: "4Gi"
        cpu: "2000m"
      limits:
        memory: "4Gi"
        cpu: "2000m"

Example kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../base

configMapGenerator:
  - name: runners-autoscaler-config
    files:
      - runners_config.yaml
    options:
      disableNameSuffixHash: true

namespace: bitbucket-runner-control-plane

commonLabels:
  app.kubernetes.io/part-of: runners-autoscaler

images:
  - name: bitbucketpipelines/runners-autoscaler
    newTag: 3.7.0 # Ensure this matches the version you checked out earlier.

patches:
  - target:
      version: v1
      kind: Secret
      name: runner-bitbucket-credentials
    patch: |-
      ### Option 1 ###
      - op: add
        path: /data/bitbucketOauthClientId
        value: "ENTER_BITBUCKET_OAUTH_CLIENT_ID_HERE_WITHIN_QUOTES"
      - op: add
        path: /data/bitbucketOauthClientSecret
        value: "ENTER_BITBUCKET_OAUTH_CLIENT_SECRET_HERE_WITHIN_QUOTES"

      ### Option 2 ###
      # - op: add
      #   path: /data/bitbucketUsername
      #   value: ""
      # - op: add
      #   path: /data/bitbucketAppPassword
      #   value: ""

  - target:
      version: v1
      kind: Deployment
      labelSelector: "inject=runners-autoscaler-envs"
    patch: |-
      ### Option 1 ###
      - op: replace
        path: /spec/template/spec/containers/0/env
        value:
          - name: BITBUCKET_OAUTH_CLIENT_ID
            valueFrom:
              secretKeyRef:
                key: bitbucketOauthClientId
                name: runner-bitbucket-credentials
          - name: BITBUCKET_OAUTH_CLIENT_SECRET
            valueFrom:
              secretKeyRef:
                key: bitbucketOauthClientSecret
                name: runner-bitbucket-credentials

      ### Option 2 ###
      # - op: replace
      #   path: /spec/template/spec/containers/0/env
      #   value:
      #     - name: BITBUCKET_USERNAME
      #       valueFrom:
      #         secretKeyRef:
      #           key: bitbucketUsername
      #           name: runner-bitbucket-credentials
      #     - name: BITBUCKET_APP_PASSWORD
      #       valueFrom:
      #         secretKeyRef:
      #           key: bitbucketAppPassword
      #           name: runner-bitbucket-credentials

Step 4: Validate the Setup

1. Check Node Status (On Workers)

sudo microk8s kubectl get nodes

2. Check Pod Status

sudo microk8s kubectl get pods -n bitbucket-runner-control-plane --field-selector=status.phase=Running

3. Inspect Runner Logs

sudo microk8s kubectl logs -f runner-controller-<pod-name> -n bitbucket-runner-control-plane

Additional Tips

• Scaling Runners: The autoscaler should handle scaling runners dynamically based on job demand. Ensure your runner configurations are correct in runners_config.yaml.
• Resource Monitoring: Use the following command to monitor resource usage and adjust configurations if necessary:

sudo microk8s kubectl top nodes

Troubleshooting

• Failed Pod Scheduling: If runners fail to schedule, check node taints and pod events.
• Logs Not Displaying: Ensure --max-log-requests is not limiting the logs excessively.

Additional Improvements

Use MicroCeph or just Ceph to manage storage that backs the MicroK8s nodes for better flexibility and scalability. Refer to the MicroCeph Multi-Node Install Guide.

[RecipeWhiz]

Setting up the Bitbucket Cloud Runner Autoscaler using GLS MicroK8s enhances your CI/CD pipeline by dynamically scaling runners to meet job demands. This integration leverages Kubernetes' orchestration capabilities, ensuring efficient resource management and streamlined operations. For a detailed guide on implementing this setup, refer to the official Bitbucket documentation.