Deploy local runner with docker-in-docker

Ferry Schoenmakers February 26, 2024

We have good experience with running some resource-heavy pipeline runners on a beefy local computer.

 

However, for easier deployment we want to use a docker-compose like below, where we're not dependent on docker being installed on the host system itself. However, we can't seem to get this to work. Has anyone did something similar already with success?

version: '2.1'

volumes:
docker_cache: {}

networks:
docker_network: {}

services:
docker:
image: docker:dind
restart: always
privileged: true
networks:
- docker_network
volumes:
- docker_cache:/var/lib/docker
bb-runner:
image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
restart: always
environment:
- WORKING_DIRECTORY=/tmp
- ACCOUNT_UUID=<> # Account UUID
- RUNNER_UUID=<> # Runner UUID
- RUNTIME_PREREQUISITES_ENABLED=true
- OAUTH_CLIENT_ID=<> # OAuth Client ID
- OAUTH_CLIENT_SECRET=<> # OAuth Client Secret
networks:
- docker_network
volumes:
- docker_cache:/var/lib/docker

We get errors like:

bb-runner-1 | [2024-02-26 12:46:02,640] Runner version: 1.559
bb-runner-1 | [2024-02-26 12:46:02,643] Runner runtime: linux-docker
bb-runner-1 | [2024-02-26 12:46:02,735] Copying Docker cli to working directory.
bb-runner-1 | [2024-02-26 12:46:02,863] An error occurred whilst retrieving the docker system info.
bb-runner-1 | java.io.FileNotFoundException: null
bb-runner-1 | at io.netty.channel.unix.Errors.newConnectException0(Errors.java:164)
bb-runner-1 | at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
bb-runner-1 | at io.netty.channel.unix.Socket.connect(Socket.java:351)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758)
bb-runner-1 | at io.netty.channel.epoll.EpollDomainSocketChannel.doConnect(EpollDomainSocketChannel.java:89)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600)
bb-runner-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632)
bb-runner-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495)
bb-runner-1 | at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51)
bb-runner-1 | at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616)
bb-runner-1 | at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978)
bb-runner-1 | at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265)
bb-runner-1 | at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264)
bb-runner-1 | at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
bb-runner-1 | at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
bb-runner-1 | at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
bb-runner-1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
bb-runner-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
bb-runner-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
bb-runner-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
bb-runner-1 | at java.base/java.lang.Thread.run(Unknown Source)

I assume the issue is related to the fact that in the oneliner you get from bitbucket to start a dockerized runner, you're also bind-mounting /var/run/docker.sock to your local system.
i.e.

docker container run -it -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID=<>  -e RUNNER_UUID=<> -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=<> -e OAUTH_CLIENT_SECRET=<> -e WORKING_DIRECTORY=/tmp --name runner-<> docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

We can't share that socket file between containers since we can only share volumes between them and sharing /var/lib does not seem a good idea?

 

Any suggestions are welcome!

0 answers

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
PERMISSIONS LEVEL
Site Admin
TAGS
AUG Leaders

Atlassian Community Events