Self hosted runner docker service fails because "docker" is a directory

Question

We use bitbucket cloud but with self hosted runners. Our jobs use the docker service. Every week or so our builds start failing with errors like `docker: Permission denied`. After some inspection in the containers, we've found this is because the "docker" command the job is trying to run is actually a directory, not an executable. Following this through the service is mounting the host path `/tmp//docker` into the containers, and on the host this sometimes becomes a directory instead of an executable file. Our solution is to stop the runner, delete the directory, and restart the runner. This then recreates `/tmp//docker` correctly and our builds work fine for about a week or more. The docker command we were given when creating the runner has a `-v /tmp:/tmp` mount. We assume the host OS does some cleanup of the /tmp directory periodically which is affecting the runner. Do we need to mount the /tmp directory, or can we use a docker volume instead to provide isolation from the host OS?

Mark G · Answer

Sorry for the delay. I'm not sure I understand your suggestion - is this is to change the bind points of the runner container without destroying it? We have no issues destroying the runner so changing the bind points is not an issue. The bind is `/tmp:/tmp`, it's the runner code that adds the docker part.

The core issue is that the code inside the runner will start new containers to run the pipeline, and (I assume) issue a docker command that binds `/tmp/<runner id>/docker` somewhere into the new temporary container. I have no control over what the code in the runner does. Even if I change the runner bind from `/tmp` to something else, when it instructs docker to start a temporary container won't it tell it to bind to `/tmp` on the host, not whatever the runner is bound to?

Janco Hoekstra · Answer

The name of the hostpath is defined in docker in the config.v2.json file and the  /docker/containers/{id}/hostconfig.json file.
Here you can rename the docker hostpath to another name, not docker. This should solve your problem.

Remember first to stop all docker containers and dockerd before making the changes and then restart them after editing.

I hope this solution fits for you.