Running Bitbucket self-hosted runners on Container-Optimized OS in Google Cloud?

[Ian Chesal]

Has anyone had any success getting self-hosted runners to launch in Docker under Google's Container-Optimized OS?

We ran a quick experiment with it, but the container has issues launching. The stack trace is substantial. But the key lines are:

2022-04-13 21:25:07,907] /tmp/libnetty_transport_native_epoll_x86_6413528779607542172002.so exists but cannot be executed even when execute permissions set; check volume for "noexec" flag; use -Dio.netty.native.workdir=[path] to set native working directory separately.

[2022-04-13 21:25:07,916] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'containerExecServiceImpl' defined in URL [jar:file:/opt/atlassian/pipelines/runner/bin/runner.jar!/com/atlassian/pipelines/runner/core/service/docker/ContainerExecServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dockerClient' defined in com.atlassian.pipelines.runner.core.configuration.spring.SpringConfigurationImpl: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.github.dockerjava.api.DockerClient]: Factory method 'dockerClient' threw exception; nested exception is java.lang.UnsatisfiedLinkError: failed to load the required native library

Maybe it's possible to bring in the native libraries that are expected here? Or maybe it's not?

The intersection of my knowledge of (CoS, GCloud, Docker, BB Runner Container) is pretty small.

Any tips from someone who's managed to make this work? If we can get CoS for runners going that's a whole lot nicer for us than a baked Debian image.

[Patrik S]

Hello @Ian Chesal ,

Welcome to Atlassian Community!

From the first error you have shared, it seems the /tmp directory used by default by runners was mounted with the noexec flag by the host OS, in this case CoS. You can list all the directories that have noexec flag by executing the following command :

mount | grep noexec

You could try removing the noexec flag from that volume and try executing the runner again, although removing that flag might have security implications.

Another option that I would personally recommend, is to change the working directory of your runner to a different folder that does not contain noexec flag.

By default, when you create a runner, you'll be given a command like the below :

docker container run -it -v /tmp:/tmp <rest of the parameters> docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

where it mounts the local directory on your machine called /tmp, as a directory inside the container also called /tmp

You can try creating another directory and use it as the tmp volume when creating a new runner container. The command would look like the following :

docker container run -it -v /home/user/runner:/tmp <rest of the parameters> docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

The above example will use  local directory /home/user/runner as the tmp folder inside the container. Please make sure the /home/user/runner directory exists in the machine where you are running the container.

I'd also recommend referring to the following documentation for additional details: 

• https://support.atlassian.com/bitbucket-cloud/docs/set-up-and-use-runners-for-linux/#Changing-the-working-directory-of-your-runner

Hope that helps. Do let us know in case you run into any issues.

Thank you @Ian Chesal

Kind regards,

Patrik S

[Ian Chesal]

This is super helpful. Thank you. I'll try it out and report back.