Bitbucket Pipelines Runners is now in open beta

[Sathish Puranika Kadinakonda]

@Justin Thomas 

I am trying to setup a runner on my AWS EKS kubernetes cluster and I could see that runner requires docker in docker to initiate the integration, which internally requires docker socket (/var/run/docker.sock). However from kuberenets perspective docker socket is not longer supported from 1.20 version onwards, hence i was unable to run the docker in docker container. This seems to be a roadblock for me to setup the runner. Without the runner running on my dedicated VPC, I cant build any of my codes. Is there any alternative to fix this issue and run on EKS?

Below is the error I am getting,

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: connect(..) failed: Connection refused: /var/run/docker.sock

Kuberenets Cluster : version 1.20

Platform : AWS EKS

Let me know if you need anymore details.

Note: I was able to run the same setup on my local kubernetes cluster which is running on 1.19 version.

Regards,

Sathish K

[János Mikó]

Hi @Justin Thomas 

I'm running the Bitbucket Pipelines Runner in our Rancher based Kubernetes cluster. It registers well, and I am able to run basic Pipelines and everything is going fine until some steps of the Pipeline has to download a large file. (In our case it was npm install)

I tried to search everywhere and I found a possible issue with network MTU.

The MTU size on the server is 1500, but the Kubernetes Network Provider is using 1450. I tried to update DIND to use the following settings:

- name: docker-in-docker
  image: docker:20.10.7-dind
  args:
    - "--mtu=1300"

Now if I attach to the Bitbucket Runner container (atlassian/bitbucket-pipelines-runner), I can see the following setting in `docker network inspect bridge`:

./docker network inspect bridge | grep mtu
"com.docker.network.driver.mtu": "1300"

 Sadly if I attach from here to a pipeline container (or the atlassian/bitbucket-pipelines-docker-daemon container) I will still see the original:

docker network inspect bridge | grep mtu
"com.docker.network.driver.mtu": "1500"

Is it possible somehow to change the MTU for the bitbucket pipelines docker daemon?

Regards,

Janos

[Nadia Begicheva]

Hi @János Mikó 

You can publish a custom dind image that accepts env variables:

# my-custom-dind-image 
FROM docker:20.10.7-dind 
ENTRYPOINT [ "sh", "-c", "dockerd-entrypoint.sh $DOCKER_OPTS" ]

and then in a pipeline configuration use it like this:

definitions: 
  services: 
     docker: 
       image: my-custom-dind-image 
       variables: 
          DOCKER_OPTS: "--mtu=1300"

 
Regards, 
Nadia

[János Mikó]

Hi @Nadia Begicheva !

Thank you for the answer! 

My DIND image is running with the proper parameter set:

/ # ps aux | grep dockerd
1 root 0:03 docker-init -- dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2376 --tlsverify --tlscacert /certs/server/ca.pem --tlscert /certs/server/cert.pem --tlskey /certs/server/key.pem --mtu=1300
62 root 3:04 dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2376 --tlsverify --tlscacert /certs/server/ca.pem --tlscert /certs/server/cert.pem --tlskey /certs/server/key.pem --mtu=1300

docker network inspect bridge | grep -i mtu
"com.docker.network.driver.mtu": "1300" 

The problem is one layer deeper. When I exec to the bitbucket-pipelines-docker-daemon container I see this:

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3ec241658084 docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable "/bin/sh /usr/local/…" 5 seconds ago Up 4 seconds (healthy) 25180cf8-96b8-5682-84fa-7fbdfda10bee_5c8ce983-6d24-4680-a5f8-7f7431e26bf4_system_auth-proxy
6944e09a77d8 docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20-prod-stable "/sbin/tini -- runit…" 5 seconds ago Up 4 seconds (unhealthy) 25180cf8-96b8-5682-84fa-7fbdfda10bee_5c8ce983-6d24-4680-a5f8-7f7431e26bf4_system_docker
0da03999d66c docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable "/bin/sh -c 'exit $(…" 6 seconds ago Up 4 seconds 25180cf8-96b8-5682-84fa-7fbdfda10bee_5c8ce983-6d24-4680-a5f8-7f7431e26bf4_clone
e1a3c394ce18 docker-hub.packages.atlassian.com/google/pause:latest "/pause" 6 seconds ago Up 4 seconds 25180cf8-96b8-5682-84fa-7fbdfda10bee_5c8ce983-6d24-4680-a5f8-7f7431e26bf4_pause
/ # docker exec -it 6944e09a77d8 sh
/ # ps aux | grep dockerd
7 root 0:00 dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --authorization-plugin=pipelines --storage-driver=overlay2 --registry-mirror http://localhost:5000 --userns-remap=default --log-level warn

And on this level I see that the network MTU is still set to 1500

docker network inspect bridge | grep -i mtu
"com.docker.network.driver.mtu": "1500"

Regards,

Janos

[Nadia Begicheva]

Hi @János Mikó

A runner spins up a dind container as part of a step setup. In your case, a default dind image was used with a default configuration. In order to use a custom docker-in-docker image, please check this documentation

I've also noticed an error in your pipeline configuration that you shared before:

1. Pipelines configuration doesn't support args property. Try to publish a custom docker image using a Dockerfile:

FROM docker:20.10.7-dind 
ENTRYPOINT [ "sh", "-c", "dockerd-entrypoint.sh $DOCKER_OPTS" ]

2. If you want to use a custom name for a docker service you should add a type: docker parameter

- name: docker-in-docker
  type: docker
  image: my-custom-dind-image # see a Dockerfile example above
  variables:
     DOCKER_OPTS: "--mtu=1300"

Hope it will help.

Regards,

Nadia 

[Justin Thomas]

@Oliver Werner We will not be charging Bitbucket Pipelines Runners on usage, there won't be any additional cost to use Runners.

[Justin Thomas]

Hi @AbdAlrahman AlaaEldeen

Bitbucket Pipeline Runners are now in GA. You can now run multiple runners on the same machine.

As for the short-term roadmap, we are currently working on Windows Runners and then we will start work on non-docker/macOS runners. Please let me know if you have any further questions.

Thanks,

Justin

[Justin Thomas]

Hi @Sathish Puranika Kadinakonda

I would suggest contacting Atlassian support, they can help you with the error.

Thanks,

Justin Thomas

[Justin Thomas]

@chris_ng You should be able to execute it on K8s. Please get in touch with Atlassian support if you run into any issues.

[Pratap Mundasada]

@Justin Thomas Could you please let me know when will it be updated to run on Windows machines?

[Justin Thomas]

@Pratap Mundasada We are currently working on Windows Runners and aiming for an EAP in November. We will soon update you on an exact date.

[Pavel Shurikov]

@Justin Thomas Do you have any links available to track non-docker runners effort or any ball-park ETA on it?

[Justin Thomas]

@Pavel Shurikov The team will work on macOS runners after shipping Windows Runners. macOS runners are non-containerized and it should be possible to use them on a Linux machine. The current plan is to ship macOS runners in the first quarter of CY22. You can follow this ticket to follow the update.

[Pavel Shurikov]

@Justin Thomas Thanks for a speedy and helpful response! I'll keep an eye on the linked ticket and for now just muck around with docker-in-docker for our runners in EKS.

[drichards-alcon]

@Pavel Shurikov  please let me know if you get DiD working on EKS, we could not and are running an EC2 with docker.

[Ingmar]

I can't see any logs from self-hosted runners. Does anyone else experience the same?

[Peng Xi]

Does it still not support linux arm64?

It has the following error

standard_init_linux.go:228: exec user process caused: exec format error

[lassian]

Hi @Peng Xi 

We do not support arm yet, please upvote this issue https://jira.atlassian.com/browse/BCLOUD-21412

@Justin Thomas do we have a rought estimate when we are looking to add Arm support?

Kind Regards,

Nathan Burrell

[Marcos De Melo Da Silva]

Hi @Justin Thomas 

You mentioned some time ago that: "one can currently start a runner in a Kubernetes cluster, auto-scaling is not yet supported."

Indeed we can get a Bitbucket runner to work just fine in our Kubernetes cluster. However, the fact that there is no auto-scaling is a roadblock for us.

Is auto-scaling in the roadmap of future improvements for self hosted runners being deployed in kubernetes?

If yes, do you have an idea when this feature will be available.

Thank you.

Regards,

Marcos.

[Justin Thomas]

@Marcos De Melo Da Silva Yes, auto-scaling is on the roadmap. We are aiming to release it in Q2 of CY2022. We will most probably be doing an early release next month, if you are interested please follow this ticket.

[Jeppe]

Are runners subject to the 1GB limit on caching docker images? Are the docker images cached on the host machine?

[plussier]

@Jeppe Rask  - Yes. I found this out the hard way. I had to drastically re-factor my containers to reduce the number of layers they have in order to keep the size down under 1GB.

[Jeppe]

@plussier Thanks for the info! But that really sucks. Why can't it just rely on the runner-machine keeping the docker image cached? We probably won't go with bitbucket pipelines then.

[plussier]

@Jeppe - I do NOT know if you're running a private runner of your own if that limit is customizable.  when relying on the Atlassian runners you are definitely limited to 1GB images...

Bitbucket pipelines IMO, is better than Jenkins. But that's it. If I had my choice, and budget, I'd move everything over to GitLab.  Their system is WAAAAY better than Atlassian's from a feature perspective as well as just general usability and integration.  GitLab is also significantly more responsive to feature requests than Atlassian is as well, IMO.

If you've got the choice to go with something better than BB pipelines, I recommend you do.

[Nicolas Esteves]

Hi @Justin Thomas

Is auto-scaling still scheduled for Q2 2022? I'm talking about this ticket.

Thank you.