How to run anything that requires privileged flag in my self-hosted runner? Trying to run litmus tes

[kalib]

I understand Bitbucket not allowing me to use the `--privileged` argument in the cloud for security reasons, but not even in my self-hosted runner? How am I supposed to run things that require that? Such as litmus puppet tests, for example?

I am trying to run:

docker run -d -it --privileged --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /tmp:exec -p 52243:22 --name docker-my-litmus-image-52243 my-litmus-service.services/litmusimage/debian:10

 Is the solution really to run docker in docker and spin up a container inside my runner container?

[Theodora Boudale · Accepted solution]

Hi @kalib,

It is possible to use the privileged flag in a runner by using an external dind image instead of the default Atlassian docker service.

An example yml file with such a definition is the following:

image: centos:7

definitions:
  services:
    docker:
      image: docker:dind

pipelines:
  default:
    - step:
        runs-on:
          - 'self.hosted'
        script: 
          - echo "this is a runner step"
          - docker run -dt --privileged centos:7 ls
        services:
          - docker

Is this something that works for you?

Kind regards,
Theodora

[kalib]

Hi @Theodora Boudale 

Have you tested this? I'm a bit confused, I am kind of new to Bitbucket pipelines.

I see you setting two images there, centos:7 at the top, and after that you set docker:dind.

I'm confused on where those go. For example, here's what I am testing.

Here is what I have at this moment:

---
definitions:
  steps:
    - step: &docker-in-docker
        image: atlassian/default-image:3
        name: Setup docker in docker
        runs-on:
          - myrunner
        script:
          - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
      services:
        - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker

When I tried what you suggested, it didn't run saying I had a format error:

---
image: ubuntu:latest
definitions:
  services:
    docker:
    image: docker:dind
  steps:
    - step: &docker-in-docker
         image: ubuntu
         name: Setup docker in docker
         runs-on:
           - my runner
         script:
           - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
        services:
          - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker

[Theodora Boudale]

Hi @kalib,

Yes, I have tested this and it works. A few notes on the last yml file you posted at the end of your previous reply:

1) The runs-on parameter must contain a self.hosted label. You can include additional labels you have added to your runner, but the self.hosted label must always be included as well.

2) I don't think that runner labels can contain spaces (I see you have added 'my runner'). Please check the labels of your runner from Repository settings > Runners

3) Another issue may be indentation. Please use the indentation as below:

image: ubuntu:latest
definitions:
  services:
    docker:
      image: docker:dind
  steps:
    - step: &docker-in-docker
        image: ubuntu
        name: Setup docker in docker
        runs-on:
          - self.hosted
          - myrunner
        script:
          - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
        services:
          - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker

I have tested the above file in my runner and it works. Indentation is usually two spaces from the previous level, except for the keywords under - step (in this case the indentation should be 4 spaces).

To answer some of the questions also regarding the images:

Pipelines (and runners) builds run in a Docker container.
For every step of your yml file, a Docker container starts, the repo is cloned, and then the commands of the step's script run in that Docker container.
Then this Docker container gets destroyed.
If there is a second step in your yml file, another Docker container starts, the repo is cloned, and the commands of the second step's script run there.

If you specify at the beginning of your yml file an image, e.g. image: ubuntu:latest or image: centos:7, every step of your build will use that image as a build container.

If you specify an image on a step level, e.g. the way you are doing here:

- step: &docker-in-docker
    image: atlassian/default-image:3

then this step will use atlassian/default-image:3 as a build container, instead of what you defined at the top.

The image definition below:

definitions:
  services:
    docker:
      image: docker:dind

This specifies what image should be used for the Docker service only.

I hope this answers your questions. If you have any other questions or if you are still running into issues, please let me know and we can look into them.

Kind regards,
Theodora

[kalib]

Oh wow, thank you very much for taking the time for this very detailed answer. It definitely helps a lot.

No, my runner doesn't have spaces in the name, I just used a random name in here to not give my runners name.

But yes, it looks like my error was in the yaml formatting, and after using your code it really worked fine with your

docker:dind option.

Thank you again for taking all the time to really explain it.

There are so many things that are for some reason still not documented. For example, none of this I was able to find in bitbucket's documentation. :/

[Theodora Boudale]

Hi @kalib,

Thank you for the update, it's good to hear that this worked and you are very welcome.

I will pass the feedback to our team so we can improve our documentation and also include information that is missing.

Please feel free to reach out if you ever need anything else!

Kind regards,
Theodora

[kalib]

Is the solution really to run docker in docker and spin up a container inside my runner container?