Welcome to the Early Access Program (EAP) for self-hosted runners in Bitbucket Pipelines. With this feature, you will be able to run builds in Pipelines on your own infrastructure. You won’t be charged for the build minutes used by your self-hosted runners. In this guide, we will walk you through the steps to set up and use runners so you can get started right away.
Prerequisites
-
a 64-Bit Linux instance with at least 8GB of RAM as a host for the runner. 2x-Steps with additional service may require more. (https://bitbucket.org/blog/support-large-builds-bitbucket-pipelines )
-
Docker needs to be installed (https://docs.docker.com/engine/install/)
- We strongly recommend disabling swap and configuring vm.swappiness. Having swap enabled can lead to non-deterministic build results in regards to memory and OOMing, meaning that sometimes enough swap is available and a build may pass while other times not enough swap is available and the same build will OOM.
Disabling swap
This will work for most Linux distributions, if the commands aren't installed you will have to install them. Consult your distributions documentation to configure swap.
1. Check if swap is enabled using the following command:
sudo swapon -sv
If you see output such as the following that means you have swap enabled.
NAME TYPE SIZE USED PRIO
/dev/sda3 partition 2G 655.2M -1
2. If swap is enabled you will have to disable it using the following process
2.1 First disable any swap using the following command
sudo swapoff -av
2.2 Than open /etc/fstab and remove any swap partitions or files that are configured
2.3 Reboot your machine
2.4 Run the following command again ensuring there is no output
sudo swapon -sv
2.5 If there is output, repeat Step 2 again ensuring all swap files are removed from /etc/fstab.
Configuring vm.swappiness
This will work for most Linux distributions, if the commands aren't installed you will have to install them. Consult your distributions documentation to configure swappiness.
1. First check the value of vm.swappiness using the following command
sudo sysctl -n vm.swappiness
If the value is anything other than 1 it means you have some swap behaviour still enabled
2. If swappiness is anything other than 1 please configure it using the following process
2.1 Open /etc/sysctl.conf and add vm.swappiness = 1 to the file on its own line
2.2 Reboot your machine
2.3 Run the following command ensuring that the output is now 1
sudo sysctl -n vm.swappiness
2.4 If there is output repeat step 2 again ensuring /etc/sysctl.conf is configured correctly
Adding a new runner in the Bitbucket UI
By adding runners to a repository, you will be able to run builds in Pipelines on your own infrastructure for specific repositories in a workspace. You can have up to 100 runners per repository.
-
To register a new runner, go to the repository you would like to create a runner for.
-
Select Repository settings from the left navigation sidebar.
-
In the Pipelines section, select Runners.
-
You will see a list of all the runners belonging to this repository and the workspace containing this repository. This will be empty if you are registering your first runner. To add a new runner, select Add runner.
-
In the Runner installation modal, you can name the new runner and assign labels. Labels allow you to schedule a step on a runner matching certain requirements. For example could you label runners with “os.linux.alpine” to target it specifically from build steps that should run on a Linux Alpine host, and a separate one named “env.prod” for runners that have a special configuration so they can deploy to a production environment. Labels can only contain lowercase, alphanumerical characters and dots. You can have up to 10 custom labels per runner, but you don’t need to add any at all, if there is no need to distinguish between runners when scheduling steps.
-
Select Next. The next page will have a pre-configured docker command that you will need to run on your intended runner host machine. It will automatically download and start the software necessary to run build steps on your host. The token in the command will automatically authenticate the host with Bitbucket. Note: Make sure you copy the command down, as you will not be able to access it again after the setup.
-
Select Next.
-
Select Finish to complete the setup. Now the new runner is listed as a runner in your repository. It will be in the unregistered state until you run the command you saved in Step 6 on your host machine. Make sure you run the command and that the state of the runner changes to ‘online’ before you try to use the runner in your Pipelines.
Adding a new workspace runner in the Bitbucket UI
By using runners within a workspace, you will be able to run builds in Pipelines on your own infrastructure for any repository in the workspace. You can have up to 100 workspace runners.
-
To register a new runner, go to the workspace you would like to create a runner for. To select a workspace, select your profile and settings avatar and select a workspace from the Recent Workspaces list or select All workspaces to open a page listing all the workspaces in which you are a member.
- Select Settings from the left navigation sidebar.
- In the Pipelines section, select Workspace runners.
-
You will see a list of all the runners belonging to this workspace. This will be empty if you are registering your first runner. To add a new runner, select Add runner.
-
To install your runner, follow the same steps 5-8 of how you would create a repository runner.
Make sure you run the command and that the state of the runner changes to ‘online’ before you try to use the runner in your Pipelines.
Using your runner in the bitbucket-pipelines.yml
To use your runner in Pipelines, add a runs-on parameter to a step, and it will run on the next available runner that has all the required labels. If all matching runners are busy, your step will wait until one becomes available again. If you don’t have any online runners in your repository or workspace that match all labels, the step will fail.
pipelines:
custom:
customPipelineWithRunnerStep:
- step:
name: Step 1
runs-on:
- 'self.hosted'
- 'os.linux.alpine'
script:
- echo "This step will run on a self hosted runner with 128 GB of memory.";
- step:
name: Step 2
script:
- echo "This step will run on Atlassian's infrastructure as usual.";
Starting your runner
Currently, we only support running 1 runner per machine
-
Start the runner using the command you got from step 6 above.
-
If this is the first time running the runner it will pull the image, if you are starting the runner again or to update the runner please always pull the runner manually before hand using the following command to ensure you always have the most up to date runner.
docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
3. If you encounter the following error please run the following command to remove the runner.
docker: Error response from daemon: Conflict. The container name "/runner" is already in use by container "c3403236e3af5962ed3a9b8771561bd2021974941cc8a89a40c6c66cecb18f53". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
docker container rm -f runner
Changing the working directory of your Runner
If you want to change the working directory that the runner uses on your host machine, add the following two flags to the docker run command when you start the runner:
docker run [all existing parameters] -v /localdirectory:/localdirectory -e WORKING_DIRECTORY=/localdirectory [runner image name]
In this command, the first value in -v parameter will be the local directory on your machine that will serve as the working directory. The second value will be the directory inside the runner. The local directory on your machine, directory inside the runner and the WORKING_DIRECTORY environment variable must all be set to the same value.
Limitations during the EAP
Note that at the moment self-hosted runners do not support every feature that you are used to from Pipelines running on Atlassian’s infrastructure. If you are using any of the following features in a step, you might not be able to run that step on a self-hosted runner:
Rest assured that we are working on these features and they will be available soon.
Leaving Feedback
We're very interested in any comments you may have around your experience with self-hosted runners! Please leave your feedback in this community group.
Need Help?
For further assistance on using the self-hosted runner:
27 comments