Bitbucket self-hosted runner failed to create shim task: OCI runtime create failed

[Yves Verrette]

We have runners that work fine and every few days we get this error and they completely stop working. 

An error occurred whilst starting container(id: 40436d4a064dbf00c0067e1ed38bc5f2ba5506a09bb517b5b12c9e00206f0a35, name: 0377585c-9fd7-5568-9fa5-99a2679abd6a_a3361eab-e986-4945-aa07-038c57d386cc_build).
com.github.dockerjava.api.exception.BadRequestException: Status 400: {"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/tmp/0377585c-9fd7-5568-9fa5-99a2679abd6a/docker\" to rootfs at \"/usr/bin/docker\": mount /tmp/0377585c-9fd7-5568-9fa5-99a2679abd6a/docker:/usr/bin/docker (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"}

Each time we have to manually remove the docker container and replace them by new ones. There seems to be a relation between time (~number of built or quantity of work) and this problem occurence. 

[Norbert Csupka · Accepted solution]

Hi Yves,

Thank you for reaching out to Atlassian Community, I'm Norbert and it's nice to meet with you!

Based on my observation the /tmp/runner_id/docker is an empty directory instead of this containing the docker executable file. We've created a document explaining this behavior: https://confluence.atlassian.com/bbkb/docker-command-not-found-error-while-running-docker-commands-in-self-hosted-runner-1206785605.html

To resolve this behavior, can we ask you to run these steps and let us know if it helps?

Solution

In this case, in order to resolve this issue, first confirm your local runner contains an empty docker folder:

$ ls -l /tmp/<docker_uuid>

Once you confirmed the directory is already there, stop the Runner and remove the local folder:

$ rm -r /tmp/<docker_uuid>/docker

Restart the Runner and confirm that in the Runners' logs you can see the following message:

Copying Docker cli to working directory.

We're looking forward to hear from you

[Yves Verrette]

Thanks Norbert that seems to be working fine. 

[Norbert Csupka]

You're most welcome Yves, I'm glad to hear I was able to help :). 

Have a great day and take care!

Norbert

[Jefferson Fermo]

hi @Norbert Csupka though this workaround works, it still keeps happening and usually on the most unfortunate times: weekends, middle of the night, holiday and is becoming very annoying. I assume the runner is using --volume to bind/mount binary 'docker' into the runner w/c creates this unexpected empty docker directory when binary docker does not exist yet. Can this be to --mount as mentioned here changed https://github.com/moby/moby/issues/13121#issuecomment-336186993 so that this does not happen.

[Norbert Csupka]

Hi @Jefferson Fermo 

I'd like to inform you that I've created a support ticket as I need a a link for a build, but I can't ask for those links here due to privacy reasons.

I'll contact you through a support ticket shortly.

[Elias Balasis]

I had to stop my runners and remove the runner folder completely , not just the "docker" sub-folder, which was wrongly mounted and shouldn't be a folder.

When I started the runners again, a "docker" executable is mounted instead.

I hope this helps.