Bitbucket Pipelines Runners is now in open beta

[Sander Mol]

Is there an ETA for the different parts currently in progress? I am currently waiting for the removal of the `max-time` restriction. Is there somewhere I can follow the status of this?

[AbdAlrahman AlaaEldeen]

@Justin Thomas 

As a Jenkins lover with a huge and long time of use, I was really surprised with this announcement on bitbucket. (specially I was going to move to GitLab for such feature)

I know it is still on beta, and there are more work to do, but would it possible to have a road map page as we used to have on Atlassian for "when" and "what" is coming.

So far I tested the current release and it is really easy and straight forward setup, but there are some points still would like to know.

You mentioned some points in your early post about having multiple runner on same host and GPU access, but during my test I found there is no parallel steps and all the parallel steps that are using the self host are queued on sequence while the steps that are running on the cloud not, and same for multiple repositories that are updated at once, is this also going to change?

[Umar Hussain]

Hi, thanks for this new feature. I'm having some problem getting my runner work. The runner is throwing error when cloning with the error :

mkfifo: /tmp/tmp/clone_result: File exists

I'm using the following docker-compose to start the runner:

version: '3'
services:
 runner:
   image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:latest
   restart: always 
   volumes: 
     - /var/run/docker.sock:/var/run/docker.sock 
     - /var/lib/docker/containers:/var/lib/docker/containers:ro 
     - workdir:/tmp 
   environment: 
     ACCOUNT_UUID: "" 
     REPOSITORY_UUID: "" 
     RUNNER_UUID: "" 
     OAUTH_CLIENT_ID: "" 
     OAUTH_CLIENT_SECRET: "" 
     WORKING_DIRECTORY: "/tmp"
volumes: workdir:

 I also tried to remove the volume and restart again but still, the issue is appearing.

I suspect that the clone container is not working correctly with this entrypoint: `"exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1)"`. Not sure though.

I also tried to map the `/tmp` of the host to container's tmp but then I get the following error for npm:

npm ERR! rofs EROFS: read-only file system, lchown '/usr/local/bin/docker'

Looking for some guidance to fix this issue.

[Martin Nicolaides]

Is it possible to use parallel steps with Runners?

[Marcus Krejpowicz]

@lassian  Thanks for finally implementing this feature. I am really looking forward to it.
One question: How far is the progress regarding the ssh keys. Seems like currently the SSH-KEYS configured in the repo are not passed to the runner instance. Is this correct? On the posted roadmap it says "ssh-keys in may". Whats the status?

Kind regards

Marcus

[Divick Kishore]

My self hosted is stuck. On doing ps I get the following output:

root     19535  0.0  0.0 113372  5216 ?        Sl   17:41   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 507c1c545a907a1b05a4b08acdc1a9fcd671ad4a44d1473faebe724afa1a6693 -addre 
root     19554  0.0  0.0   1552   248 ?        Ss   17:41   0:00  \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root     19591  0.0  0.0   1552   116 ?        S    17:41   0:00      \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root     19592  0.0  0.0   1484   252 ?        S    17:41   0:00          \_ /bin/cat /tmp/tmp/clone_result

 I am not sure why it is stuck at this?

Not mounting /tmp into docker doesn't seem to make any difference. Moreover I see that when the job is launched, it seems to run a few more containers:

CONTAINER ID   IMAGE                                                                                           COMMAND                  CREATED              STATUS                        PO
RTS     NAMES 
e34134bb2f12   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable   "/bin/sh /usr/local/…"   About a minute ago   Up About a minute (healthy)     
       system-auth-proxy 
f0fe3a7fed59   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable   "/bin/sh -c 'exit $(…"   About a minute ago   Up About a minute               
       clone 
41d80ab0543c   google/pause:latest                                                                             "/pause"                 About a minute ago   Up About a minute               
       pause 
0c79cfe21b71   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner                   "/bin/sh -c -x ./ent…"   16 minutes ago       Up 16 minutes                   
       rol-dev-runner

rol-dev-runner is my own runner and rest 3 are spawned by bitbucket-runner. Doing a ps on them I see this:

root     28680  0.0  0.1 113372  9772 ?        Sl   19:25   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0c79cfe21b71b2a15c0d0c05feaa9796e15d9368a269cf8d7362cca664c49aed -addre 
root     28701  0.0  0.0   2376   648 ?        Ss+  19:25   0:00  \_ /bin/sh -c -x ./entrypoint.sh 
root     28737  0.0  0.0   2376   656 ?        S+   19:25   0:00      \_ /bin/sh ./entrypoint.sh 
root     28738  2.2  5.9 5861128 475856 ?      Sl+  19:25   0:24          \_ java -jar -Dbitbucket.pipelines.runner.account.uuid={...removed} -Dbitbucket.pipelines 
root      8932  0.0  0.0 111708  5200 ?        Sl   19:39   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 41d80ab0543c3f76c23fd0c582af9bf84634a88eb0f94c5c4e3c6e257d8b40d3 -addre 
root      8953  0.0  0.0 266736   248 ?        Ssl  19:39   0:00  \_ /pause 
root      9003  0.0  0.0 113372  5572 ?        Sl   19:40   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f0fe3a7fed592cfb5966aba5759c75fc1c44b52296085e6257a3fcf44e65a975 -addre 
root      9023  0.0  0.0   1552   248 ?        Ss   19:40   0:00  \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root      9111  0.0  0.0   1552   116 ?        S    19:40   0:00      \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root      9112  0.0  0.0   1484   252 ?        S    19:40   0:00          \_ /bin/cat /tmp/tmp/clone_result 
root      9045  0.8  0.0 113372  6588 ?        Sl   19:40   0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id e34134bb2f129a8b2cbc1a42720e01c5be535ae9190ed3fed72e101a0b3cf0e9 -addre 
root      9077  0.0  0.1  26024 14992 ?        Ss   19:40   0:00  \_ nginx: master process nginx -g daemon off; 
100       9129  0.0  0.1  26260 13416 ?        S    19:40   0:00      \_ nginx: worker process

I am not sure why it is stuck.

Any help folks?

Best,

[itsmesunil]

This is great. Is there a possibility where we can use one runner for multiple repos?

Like registering a group runner or something

[lassian]

Hi @itsmesunil

We are currently working on the ability to allow users to add runners at the workspace level to be shared amongst the repositories in the workspace.

Keep an eye out for this feature in the coming weeks :)

Kind Regards,
Nathan

[itsmesunil]

@lassian Thanks

[Divick Kishore]

My self hosted is stuck. On doing ps I get the following output:

root     19535  0.0  0.0 113372  5216 ?        Sl   17:41   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 507c1c545a907a1b05a4b08acdc1a9fcd671ad4a44d1473faebe724afa1a6693 -addre 
root     19554  0.0  0.0   1552   248 ?        Ss   17:41   0:00  \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root     19591  0.0  0.0   1552   116 ?        S    17:41   0:00      \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root     19592  0.0  0.0   1484   252 ?        S    17:41   0:00          \_ /bin/cat /tmp/tmp/clone_result

 I am not sure why it is stuck at this?

Not mounting /tmp into docker doesn't seem to make any difference. Moreover I see that when the job is launched, it seems to run a few more containers:

CONTAINER ID   IMAGE                                                                                           COMMAND                  CREATED              STATUS                        PO
RTS     NAMES 
e34134bb2f12   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable   "/bin/sh /usr/local/…"   About a minute ago   Up About a minute (healthy)     
       system-auth-proxy 
f0fe3a7fed59   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable   "/bin/sh -c 'exit $(…"   About a minute ago   Up About a minute               
       clone 
41d80ab0543c   google/pause:latest                                                                             "/pause"                 About a minute ago   Up About a minute               
       pause 
0c79cfe21b71   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner                   "/bin/sh -c -x ./ent…"   16 minutes ago       Up 16 minutes                   
       rol-dev-runner

rol-dev-runner is my own runner and rest 3 are spawned by bitbucket-runner. Doing a ps on them I see this:

root     28680  0.0  0.1 113372  9772 ?        Sl   19:25   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0c79cfe21b71b2a15c0d0c05feaa9796e15d9368a269cf8d7362cca664c49aed -addre 
root     28701  0.0  0.0   2376   648 ?        Ss+  19:25   0:00  \_ /bin/sh -c -x ./entrypoint.sh 
root     28737  0.0  0.0   2376   656 ?        S+   19:25   0:00      \_ /bin/sh ./entrypoint.sh 
root     28738  2.2  5.9 5861128 475856 ?      Sl+  19:25   0:24          \_ java -jar -Dbitbucket.pipelines.runner.account.uuid={...removed} -Dbitbucket.pipelines 
root      8932  0.0  0.0 111708  5200 ?        Sl   19:39   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 41d80ab0543c3f76c23fd0c582af9bf84634a88eb0f94c5c4e3c6e257d8b40d3 -addre 
root      8953  0.0  0.0 266736   248 ?        Ssl  19:39   0:00  \_ /pause 
root      9003  0.0  0.0 113372  5572 ?        Sl   19:40   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f0fe3a7fed592cfb5966aba5759c75fc1c44b52296085e6257a3fcf44e65a975 -addre 
root      9023  0.0  0.0   1552   248 ?        Ss   19:40   0:00  \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root      9111  0.0  0.0   1552   116 ?        S    19:40   0:00      \_ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1) 
root      9112  0.0  0.0   1484   252 ?        S    19:40   0:00          \_ /bin/cat /tmp/tmp/clone_result 
root      9045  0.8  0.0 113372  6588 ?        Sl   19:40   0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id e34134bb2f129a8b2cbc1a42720e01c5be535ae9190ed3fed72e101a0b3cf0e9 -addre 
root      9077  0.0  0.1  26024 14992 ?        Ss   19:40   0:00  \_ nginx: master process nginx -g daemon off; 
100       9129  0.0  0.1  26260 13416 ?        S    19:40   0:00      \_ nginx: worker process

I am not sure why it is stuck.

Any help folks?

[Divick Kishore]

My self hosted is stuck. On doing ps I get the following output:

root     19535  0.0  0.0 113372  5216 ?        Sl   17:41   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 507c1c545a907a1b05a4b08acdc1a9fcd671ad4a44d1473faebe724afa1a6693 -addre
root     19554  0.0  0.0   1552   248 ?        Ss   17:41   0:00  _ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1)
root     19591  0.0  0.0   1552   116 ?        S    17:41   0:00      _ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1)
root     19592  0.0  0.0   1484   252 ?        S    17:41   0:00          _ /bin/cat /tmp/tmp/clone_result
 I am not sure why it is stuck at this?

Not mounting /tmp into docker doesn't seem to make any difference. Moreover I see that when the job is launched, it seems to run a few more containers:

CONTAINER ID   IMAGE                                                                                           COMMAND                  CREATED              STATUS                        PO
RTS     NAMES
e34134bb2f12   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable   "/bin/sh /usr/local/…"   About a minute ago   Up About a minute (healthy)     
       system-auth-proxy
f0fe3a7fed59   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable   "/bin/sh -c 'exit $(…"   About a minute ago   Up About a minute               
       clone
41d80ab0543c   google/pause:latest                                                                             "/pause"                 About a minute ago   Up About a minute               
       pause
0c79cfe21b71   docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner                   "/bin/sh -c -x ./ent…"   16 minutes ago       Up 16 minutes                   
       rol-dev-runner

rol-dev-runner is my own runner and rest 3 are spawned by bitbucket-runner. Doing a ps on them I see this:

root     28680  0.0  0.1 113372  9772 ?        Sl   19:25   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0c79cfe21b71b2a15c0d0c05feaa9796e15d9368a269cf8d7362cca664c49aed -addre
root     28701  0.0  0.0   2376   648 ?        Ss+  19:25   0:00  _ /bin/sh -c -x ./entrypoint.sh
root     28737  0.0  0.0   2376   656 ?        S+   19:25   0:00      _ /bin/sh ./entrypoint.sh
root     28738  2.2  5.9 5861128 475856 ?      Sl+  19:25   0:24          _ java -jar -Dbitbucket.pipelines.runner.account.uuid={...removed} -Dbitbucket.pipelines
root      8932  0.0  0.0 111708  5200 ?        Sl   19:39   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 41d80ab0543c3f76c23fd0c582af9bf84634a88eb0f94c5c4e3c6e257d8b40d3 -addre
root      8953  0.0  0.0 266736   248 ?        Ssl  19:39   0:00  _ /pause
root      9003  0.0  0.0 113372  5572 ?        Sl   19:40   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f0fe3a7fed592cfb5966aba5759c75fc1c44b52296085e6257a3fcf44e65a975 -addre
root      9023  0.0  0.0   1552   248 ?        Ss   19:40   0:00  _ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1)
root      9111  0.0  0.0   1552   116 ?        S    19:40   0:00      _ /bin/sh -c exit $( (/usr/bin/mkfifo /tmp/tmp/clone_result && /bin/cat /tmp/tmp/clone_result) || /bin/echo 1)
root      9112  0.0  0.0   1484   252 ?        S    19:40   0:00          _ /bin/cat /tmp/tmp/clone_result
root      9045  0.8  0.0 113372  6588 ?        Sl   19:40   0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id e34134bb2f129a8b2cbc1a42720e01c5be535ae9190ed3fed72e101a0b3cf0e9 -addre
root      9077  0.0  0.1  26024 14992 ?        Ss   19:40   0:00  _ nginx: master process nginx -g daemon off;
100       9129  0.0  0.1  26260 13416 ?        S    19:40   0:00      _ nginx: worker process

I am not sure why it is stuck.

Any help folks?

[Bharadwaj Sattaru]

We created a self hosted runner to run the pipelines. But we are not able to access the internal links (resources) from docker containers created by runner. We are able to access it, If we add host entry inside the container. But here the containers are creating on demand based on pipeline trigger. So can someone please help me to resolve this issue?

[mwatson]

For updates please follow BCLOUD-16995 

[Vytenis Ščiukas]

Hello, Ive managed to run multiple runners on one host using docker dind.
Ive put an example of how to run this:

First runner

docker network create my-first-runner-network
docker volume create my-first-runner-certs-ca
docker volume create my-first-runner-certs-client
docker run --restart always --privileged --name my-first-runner-dind -d --network my-first-runner-network --network-alias docker -e DOCKER_TLS_CERTDIR=/certs -v my-first-runner-certs-ca:/certs/ca -v my-first-runner-certs-client:/certs/client docker:dind
docker run --restart always -d --network my-first-runner-network -e DOCKER_TLS_CERTDIR=/certs -v my-first-runner-certs-client:/certs/client:ro --name my-first-runner docker:latest run -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID={XXX} -e REPOSITORY_UUID={XXX} -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=XXX -e OAUTH_CLIENT_SECRET=XXX -e WORKING_DIRECTORY=/tmp docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

Second runner

docker network create my-second-runner-network
docker volume create my-second-runner-certs-ca
docker volume create my-second-runner-certs-client
docker run --restart always --privileged --name my-second-runner-dind -d --network my-second-runner-network --network-alias docker -e DOCKER_TLS_CERTDIR=/certs -v my-second-runner-certs-ca:/certs/ca -v my-second-runner-certs-client:/certs/client docker:dind
docker run --restart always -d --network my-second-runner-network -e DOCKER_TLS_CERTDIR=/certs -v my-second-runner-certs-client:/certs/client:ro --name my-second-runner docker:latest run -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID={XXX} -e REPOSITORY_UUID={XXX} -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=XXX -e OAUTH_CLIENT_SECRET=XXX -e WORKING_DIRECTORY=/tmp docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

As you can see on last command for each runner ive used same command that bitbucket provides just docker container removed at the start, -it option and --name argument, because with named containers will be unable to autostart.

Known issues: New container of a runner is being created every time server is restarted and old ones are kept.

You can clear old containers using:

docker container exec -it my-first-runner-dind docker system prune
docker container exec -it my-second-runner-dind docker system prune

[alexey]

it would be great to have ability to define default runner...

[lassian]

Hi @alexey

Can you describe what you mean by default runner?

You mean a fallback if none are found?

If so that can be problematic as labels are used to match a runner that can run that step, if it falls back to some default it may not be able to run the step as the default runner may not have the capabilities that the runner that matches the labels on the step has.

If its that you want to run on any runner regardless of labels you can simply use the built in label 'self.hosted' than it'll run on any runner.

Kind Regards,

Nathan

[Alexey Wasilyev]

Currently if no runs-on specified step is running on internal bitbucket runner. I would like to be able to change this behavior and choose per project/workspace what runner to use if there is no runs-on parameter. 

Motivation: we have a lot of repositories and would like them on to be compiled/executed on our runner. Now it requires changes for all steps in all bitbucket pipelines 

[lassian]

Hi @alexey

So thats by design as no runs-on is backwards compatible with all existing steps which means to use our cloud as you can mix and match cloud runners and self hosted runners within a pipeline based on lack of no runs-on as you can share caches, artifacts etc between them.

I can see your use case but you would like to enforce no cloud runners at all in that workspace/repository? And have the default in that case be whatever runner you have setup?

Kind Regards,
Nathan

[Oliver Werner]

Hi @Justin Thomas ,
I understand that the self-hosted Bitbucket Pipeline runners will not be charged for their build minutes. Will there be a licensing system that charges the users for the amount of self-hosted Runners they use (like Bamboo Server uses at the moment)? Can you give some more insights into the prospected costs involved with using self-hosted Runners?

Thanks in advance!

[chris_ng]

I managed to get a pod running with the 2 containers; however, setting the "runs-on" attribute to use the runner results in a volume mount error:

Status 500: {"message":"error creating aufs mount to /var/lib/docker/aufs/mnt/b443f67850b3ec86a884864e7fec319d482124fd4a7592461577d085c469286b-init: mount target=/var/lib/docker/aufs/mnt/b443f67850b3ec86a884864e7fec319d482124fd4a7592461577d085c469286b-init data=br:/var/lib/docker/aufs/diff/b443f67850b3ec86a884864e7fec319d482124fd4a7592461577d085c469286b-init=rw:/var/lib/docker/aufs/diff/d97807a81c540ec15035784f2b5bad8cefa8b889c2d008db873b18a6a7b8b1ce=ro+wh:/var/lib/docker/aufs/diff/1ebbacb2bbf0f1616505d881623badb3be59929cd7fc55f8e797b2b04fafaad2=ro+wh:/var/lib/docker/aufs/diff/76da949d3eaa7ac3aea6fc1cbc95f04d8cde204294232d521f0df9d527715b84=ro+wh,dio,xino=/dev/shm/aufs.xino: invalid argument"}

Anyone else having a similar issue?

[Renato Baudouin]

Could the runner be used to deploy an artifact onPremise?

[Vytenis Ščiukas]

@Renato Baudouindepends on how your server is accessible. Self hosted runner could do almost anything a docker container can do with host machine.

[Renato Baudouin]

@Vytenis Ščiukas Thank you very much for your response. If I would like to deploy my artifact to the same server where my runner is hosted, how should I set up my deployment steps?

[Vytenis Ščiukas]

@Renato Baudouin based on docs https://support.atlassian.com/bitbucket-cloud/docs/set-up-and-use-runners-for-linux/

you can use docker volumes, so localdirectory on these docs are your local path, what you need to test is what happens to the files after runner is terminated. If files is being destroyed you can always copy them to other location than working dir, you can also mount as many volumes as you want and deploy those files from working dir to any volume you want.

[chris_ng]

@Justin Thomas has anyone gotten the runner to work on a kubernetes cluster?