Getting started with Bitbucket Pipelines Runners

Welcome to the Early Access Program (EAP) for self-hosted runners in Bitbucket Pipelines. With this feature, you will be able to run builds in Pipelines on your own infrastructure. You won’t be charged for the build minutes used by your self-hosted runners. In this guide, we will walk you through the steps to set up and use runners so you can get started right away.

Prerequisites

  • a 64-Bit Linux instance with at least 8GB of RAM as a host for the runner. 2x-Steps with additional service may require more. (https://bitbucket.org/blog/support-large-builds-bitbucket-pipelines )

  • Docker needs to be installed (https://docs.docker.com/engine/install/)

  • We strongly recommend disabling swap and configuring vm.swappiness. Having swap enabled can lead to non-deterministic build results in regards to memory and OOMing, meaning that sometimes enough swap is available and a build may pass while other times not enough swap is available and the same build will OOM.

Disabling swap

This will work for most Linux distributions, if the commands aren't installed you will have to install them. Consult your distributions documentation to configure swap.

1. Check if swap is enabled using the following command:

sudo swapon -sv

If you see output such as the following that means you have swap enabled.

NAME      TYPE      SIZE   USED PRIO
/dev/sda3 partition   2G 655.2M   -1

2. If swap is enabled you will have to disable it using the following process

2.1 First disable any swap using the following command

sudo swapoff -av

2.2 Than open /etc/fstab and remove any swap partitions or files that are configured

2.3 Reboot your machine

2.4 Run the following command again ensuring there is no output

sudo swapon -sv

2.5 If there is output, repeat Step 2 again ensuring all swap files are removed from /etc/fstab.

Configuring vm.swappiness

This will work for most Linux distributions, if the commands aren't installed you will have to install them. Consult your distributions documentation to configure swappiness.

1. First check the value of vm.swappiness using the following command

sudo sysctl -n vm.swappiness

If the value is anything other than 1 it means you have some swap behaviour still enabled

2. If swappiness is anything other than 1 please configure it using the following process

2.1 Open /etc/sysctl.conf and add vm.swappiness = 1 to the file on its own line

2.2 Reboot your machine

2.3 Run the following command ensuring that the output is now 1

sudo sysctl -n vm.swappiness

2.4 If there is output repeat step 2 again ensuring /etc/sysctl.conf is configured correctly

Adding a new runner in the Bitbucket UI

By adding runners to a repository, you will be able to run builds in Pipelines on your own infrastructure for specific repositories in a workspace. You can have up to 100 runners per repository.

  1. To register a new runner, go to the repository you would like to create a runner for.

  2. Select Repository settings from the left navigation sidebar.

    1.png
  3. In the Pipelines section, select Runners.

    2.png
  4. You will see a list of all the runners belonging to this repository and the workspace containing this repository. This will be empty if you are registering your first runner. To add a new runner, select Add runner.

  5. In the Runner installation modal, you can name the new runner and assign labels. Labels allow you to schedule a step on a runner matching certain requirements. For example could you label runners with “os.linux.alpine” to target it specifically from build steps that should run on a Linux Alpine host, and a separate one named “env.prod” for runners that have a special configuration so they can deploy to a production environment. Labels can only contain lowercase, alphanumerical characters and dots. You can have up to 10 custom labels per runner, but you don’t need to add any at all, if there is no need to distinguish between runners when scheduling steps.

    image-20210301-000706.png
  6. Select Next. The next page will have a pre-configured docker command that you will need to run on your intended runner host machine. It will automatically download and start the software necessary to run build steps on your host. The token in the command will automatically authenticate the host with Bitbucket. Note: Make sure you copy the command down, as you will not be able to access it again after the setup.

    image-20210223-055120.png
  7. Select Next.

  8. Select Finish to complete the setup. Now the new runner is listed as a runner in your repository. It will be in the unregistered state until you run the command you saved in Step 6 on your host machine. Make sure you run the command and that the state of the runner changes to ‘online’ before you try to use the runner in your Pipelines.

 

Adding a new workspace runner in the Bitbucket UI

By using runners within a workspace, you will be able to run builds in Pipelines on your own infrastructure for any repository in the workspace. You can have up to 100 workspace runners.

  1. To register a new runner, go to the workspace you would like to create a runner for. To select a workspace, select your profile and settings avatar and select a workspace from the Recent Workspaces list or select All workspaces to open a page listing all the workspaces in which you are a member.

  2. Select Settings from the left navigation sidebar.Screen Shot 2021-07-09 at 2.37.08 pm.png
  3. In the Pipelines section, select Workspace runners.Screen Shot 2021-07-09 at 2.52.16 pm.png
  4. You will see a list of all the runners belonging to this workspace. This will be empty if you are registering your first runner. To add a new runner, select Add runner.

  5. To install your runner, follow the same steps 5-8 of how you would create a repository runner.

 

Make sure you run the command and that the state of the runner changes to ‘online’ before you try to use the runner in your Pipelines.

 

Using your runner in the bitbucket-pipelines.yml

To use your runner in Pipelines, add a runs-on parameter to a step, and it will run on the next available runner that has all the required labels. If all matching runners are busy, your step will wait until one becomes available again. If you don’t have any online runners in your repository or workspace that match all labels, the step will fail.

pipelines:
  custom:
    customPipelineWithRunnerStep:
      - step:
          name: Step 1
          runs-on: 
            - 'self.hosted'
            - 'os.linux.alpine'
          script:
            - echo "This step will run on a self hosted runner with 128 GB of memory.";
      - step:
          name: Step 2
          script:
            - echo "This step will run on Atlassian's infrastructure as usual.";

Starting your runner

Currently, we only support running 1 runner per machine

  1. Start the runner using the command you got from step 6 above.

  2. If this is the first time running the runner it will pull the image, if you are starting the runner again or to update the runner please always pull the runner manually before hand using the following command to ensure you always have the most up to date runner.

docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

3. If you encounter the following error please run the following command to remove the runner.

docker: Error response from daemon: Conflict. The container name "/runner" is already in use by container "c3403236e3af5962ed3a9b8771561bd2021974941cc8a89a40c6c66cecb18f53". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
docker container rm -f runner

Changing the working directory of your Runner

If you want to change the working directory that the runner uses on your host machine, add the following two flags to the docker run command when you start the runner:

docker run [all existing parameters] -v /localdirectory:/localdirectory -e WORKING_DIRECTORY=/localdirectory [runner image name] 

 

In this command, the first value in -v parameter will be the local directory on your machine that will serve as the working directory. The second value will be the directory inside the runner. The local directory on your machine, directory inside the runner and the WORKING_DIRECTORY environment variable must all be set to the same value.

Limitations during the EAP

Note that at the moment self-hosted runners do not support every feature that you are used to from Pipelines running on Atlassian’s infrastructure. If you are using any of the following features in a step, you might not be able to run that step on a self-hosted runner:

Rest assured that we are working on these features and they will be available soon.

Leaving Feedback

We're very interested in any comments you may have around your experience with self-hosted runners! Please leave your feedback in this community group. 

Need Help?

For further assistance on using the self-hosted runner:

  • Post your question in this community group.

27 comments

Comment

Log in or Sign up to comment
Akshay Sharma March 5, 2021

Hi Team,

i configured the runner and tried running the pipeline. but it is giving below error in runner logs.

[2021-03-05 14:39:50,173] Completing step with result Result{status=ERROR, error=Some(Error{key='runner.bitbucket-pipelines.clone-container-failure', message='We couldn't clone the repository. Try rerunning the pipeline.', arguments={}})}.
[2021-03-05 14:39:50,435] Waiting for next step.

and in pipeline UI below error is showing.

Error

We couldn't clone the repository. Try rerunning the pipeline.

 

Please suggest.

Thanks,

Akshay

daniego March 7, 2021

Hi,

Unfortunately I'm unable to create any runner as from repo as the page at step 4 says that the feature is coming soon.

Is there anything my side I've missed to join the EAP?

 

Thanks

Like # people like this
Francisco Gonzalez March 8, 2021

Hi team! Glad to see runners finally. I have a couple of questions.

 

Can we expect the ability to add runners at a team/account level when this is released?

Will we be able to run more than one runner per machine?

 

Cheers,

Francisco.

Like # people like this
Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 10, 2021

@Francisco Gonzalez Thanks for trying the runners. Workspace runners and multiple runners per machine are not planned for the GA release. Workspace runners is definitely something that we want to look at after the GA.

Can you please explain a bit more about your use case for running multiple runners per machine?

Francisco Gonzalez March 11, 2021

@Justin Thomas our developers currently build and deploy their apps with Bamboo. Since Bamboo is now EOL, we are hoping to be able to move those builds from Bamboo to Pipelines. We need Runners in order to access servers inside our VPC, such as Artifactory. We have 12 agents in our Bamboo instance. This guarantees a low wait time for our developers.

So, we need to be able to provide a pool of 12 agents or runners. We could do this with 12 EC2 instances of course, but it would be nice to be able to group some of them on the same instance. This way, one single agent wouldn't be restricted to the memory of a single, smaller instance.

This is also why we need workspace runners. Our development teams share this pool of 12 agents. It would be a waste of time and resources to have to create one runner (and EC2 instance) per repository, given the number of repositories we have (+500).

Like # people like this
Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 11, 2021

Thanks, @Francisco Gonzalez for explaining the use case, it helps us prioritize the feature.

Vladislav Zhukov March 11, 2021

Hello!

Unfortunately, I can't add runners as it's still unavailable.

1.png

Like daniego likes this
Gabriel Cyr March 12, 2021

@Vladislav Zhukov i have the same issue weirdly enough.

Like daniego likes this
Nitin Goyal March 14, 2021

@Justin Thomasdoes Self Hosted Runners support ARM build?

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 14, 2021

@Nitin Goyal The EAP doesn't support ARM, can you please explain the use case? this would help us prioritize the feature. 

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 14, 2021

@Gabriel Cyr @Vladislav Zhukov We suspect the issue is caused by caching. Can you clear the cache/cookies and try again?

Like Vladislav Zhukov likes this
Nitin Goyal March 14, 2021

@Justin Thomas We are running our workloads on AWS using EKS. We would like to use Arm based Instance but as bitbucket-pipelines doesn't support ARM based build it is restricting us to using arm based instances.

Other CI/CD system provide arm based docker build except bitbucket. :(

Like Justin Thomas likes this
Vladislav Zhukov March 15, 2021

@Justin Thomas It works!

Thanks!

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 15, 2021

Thanks, @Nitin Goyal for explaining the use case.

Mark Tsyrulnyk March 19, 2021

Is there a way to run on sel-fhosted runners with fallback to bitbucket host if no runners are available?

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 22, 2021

@Mark Tsyrulnyk Can you explain a bit more about your use case and what problem you are trying to solve? Currently, we don't have any plans to add a fallback functionality between self-hosted and Bitbucket runners.

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 22, 2021
Mark Tsyrulnyk March 23, 2021

@Justin Thomas the idea is to run pipeline when possible on local runners (as they executed on more powerful machine that bitbucket pipelines) but when no local runners available, fallback execution to bitbucket pipeline runner ensure that pipelines will be executed

e. g.

runs-on:
- 'self.hosted.ubuntu'
- 'self.hosted.alpine'
- 'bitbucket.hosted'

so if neither self.hosted.ubuntu nor self.hosted.alpine available, pipeline will be executed on bitbucket host (default host for pipelines)

Like # people like this
Artsiom Zhurbila March 25, 2021

faced with a situation when runner doesn't clean /tmp/tmp properly sometimes

bb-runner-error.png

Artsiom Zhurbila March 25, 2021

> [2021-03-25 09:07:23,395] Polling for runner after 0 seconds and then every 30 seconds.

is it possible to make it configurable or decrease it to "every 5 seconds"?

Artsiom Zhurbila March 26, 2021

Does runner support to mount host dir inside container (for shared files/cache)?


inspecting runner docker container found only

```
{
"Type": "bind",
"Source": "/tmp/tmp",
"Destination": "/tmp/tmp",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
```

found only this place to set gradle cahce to volume which is mounted to host

`export GRADLE_USER_HOME=/tmp/tmp/.gradle`

but this tmp dir clean up at the end of each CI run

do you have other possibility to attach host dir inside runner container?

like - <host>:<ci container>

volumes:
- /caches/gradle:/home/gradle/.gradle/caches/

lassian
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 28, 2021

HI Artsiom,

"faced with a situation when runner doesn't clean /tmp/tmp properly sometimes"

Can you please provide the runners logs for the time at which you execute the step so we can further diagnose.

Also can you confirm if you were running more than one runner on that machine at the time as we dont support that currently and they both clash with sharing tmp.

"is it possible to make it configurable or decrease it to "every 5 seconds"

These is no need to decrease this the runner listens to new steps on a websocket aswell which within a 1s will pickup the step assigned to it in most cases the polling is there for a fallback when the runner is either:

a) offline when it is scheduled

b) the websocket fails.

"Does runner support to mount host dir inside container (for shared files/cache)?"

No it does not.

Kind Regards,
Nathan Burrell

Like Artsiom Zhurbila likes this
Artsiom Zhurbila March 29, 2021

One of our goal to use on-site runners: has memory more than default bitbucket CI sizex2

for example we have 1 VM with 32 GB of ram and we want to utilize all 32 GB

but on-site runner still has limit of 8 GB

```
docker inspect eb1c5a5f129c | grep Mem
"Memory": 8589934592,
"CpusetMems": "",
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": -1,
"MemorySwappiness": 0,
```

Is it possible to change such behavior?

Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
March 30, 2021

@Artsiom Zhurbila The current self-hosted runner has the same behavior/limitations as the cloud runner, but we will be relaxing the limitations in the near future.

TAGS
AUG Leaders

Atlassian Community Events