Deploy local runner with docker-in-docker

[Ferry Schoenmakers]

We have good experience with running some resource-heavy pipeline runners on a beefy local computer.

However, for easier deployment we want to use a docker-compose like below, where we're not dependent on docker being installed on the host system itself. However, we can't seem to get this to work. Has anyone did something similar already with success?

version: '2.1'

volumes:
  docker_cache: {}

networks:
  docker_network: {}

services:
  docker:
    image: docker:dind
    restart: always
    privileged: true
    networks:
      - docker_network
    volumes:
      - docker_cache:/var/lib/docker
  bb-runner:
    image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
    restart: always
    environment:
      - WORKING_DIRECTORY=/tmp
      - ACCOUNT_UUID=<> # Account UUID
      - RUNNER_UUID=<> # Runner UUID
      - RUNTIME_PREREQUISITES_ENABLED=true
      - OAUTH_CLIENT_ID=<> # OAuth Client ID
      - OAUTH_CLIENT_SECRET=<> # OAuth Client Secret
    networks:
      - docker_network
    volumes:
      - docker_cache:/var/lib/docker

We get errors like:

bb-runner-1 | [2024-02-26 12:46:02,640] Runner version: 1.559
bb-runner-1 | [2024-02-26 12:46:02,643] Runner runtime: linux-docker
bb-runner-1 | [2024-02-26 12:46:02,735] Copying Docker cli to working directory.
bb-runner-1 | [2024-02-26 12:46:02,863] An error occurred whilst retrieving the docker system info.
bb-runner-1 | java.io.FileNotFoundException: null
bb-runner-1 | at io.netty.channel.unix.Errors.newConnectException0(Errors.java:164)
bb-runner-1 | at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
bb-runner-1 | at io.netty.channel.unix.Socket.connect(Socket.java:351)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758)
bb-runner-1 | at io.netty.channel.epoll.EpollDomainSocketChannel.doConnect(EpollDomainSocketChannel.java:89)
bb-runner-1 | at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600)
bb-runner-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632)
bb-runner-1 | at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495)
bb-runner-1 | at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51)
bb-runner-1 | at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632)
bb-runner-1 | at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616)
bb-runner-1 | at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978)
bb-runner-1 | at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265)
bb-runner-1 | at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264)
bb-runner-1 | at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
bb-runner-1 | at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
bb-runner-1 | at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
bb-runner-1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
bb-runner-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
bb-runner-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
bb-runner-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
bb-runner-1 | at java.base/java.lang.Thread.run(Unknown Source)

I assume the issue is related to the fact that in the oneliner you get from bitbucket to start a dockerized runner, you're also bind-mounting /var/run/docker.sock to your local system.
i.e.

docker container run -it -v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro -e ACCOUNT_UUID=<>  -e RUNNER_UUID=<> -e RUNTIME_PREREQUISITES_ENABLED=true -e OAUTH_CLIENT_ID=<> -e OAUTH_CLIENT_SECRET=<> -e WORKING_DIRECTORY=/tmp --name runner-<> docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1

We can't share that socket file between containers since we can only share volumes between them and sharing /var/lib does not seem a good idea?

Any suggestions are welcome!