We have runners that work fine and every few days we get this error and they completely stop working.
An error occurred whilst starting container(id: 40436d4a064dbf00c0067e1ed38bc5f2ba5506a09bb517b5b12c9e00206f0a35, name: 0377585c-9fd7-5568-9fa5-99a2679abd6a_a3361eab-e986-4945-aa07-038c57d386cc_build).
com.github.dockerjava.api.exception.BadRequestException: Status 400: {"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/tmp/0377585c-9fd7-5568-9fa5-99a2679abd6a/docker\" to rootfs at \"/usr/bin/docker\": mount /tmp/0377585c-9fd7-5568-9fa5-99a2679abd6a/docker:/usr/bin/docker (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"}
Each time we have to manually remove the docker container and replace them by new ones. There seems to be a relation between time (~number of built or quantity of work) and this problem occurence.
Hi Yves,
Thank you for reaching out to Atlassian Community, I'm Norbert and it's nice to meet with you!
Based on my observation the /tmp/runner_id/docker is an empty directory instead of this containing the docker executable file. We've created a document explaining this behavior: https://confluence.atlassian.com/bbkb/docker-command-not-found-error-while-running-docker-commands-in-self-hosted-runner-1206785605.html
To resolve this behavior, can we ask you to run these steps and let us know if it helps?
In this case, in order to resolve this issue, first confirm your local runner contains an empty docker folder:
$ ls -l /tmp/<docker_uuid>
Once you confirmed the directory is already there, stop the Runner and remove the local folder:
$ rm -r /tmp/<docker_uuid>/docker
Restart the Runner and confirm that in the Runners' logs you can see the following message:
Copying Docker cli to working directory.
We're looking forward to hear from you
Thanks Norbert that seems to be working fine.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You're most welcome Yves, I'm glad to hear I was able to help :).
Have a great day and take care!
Norbert
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
hi @Norbert Csupka though this workaround works, it still keeps happening and usually on the most unfortunate times: weekends, middle of the night, holiday and is becoming very annoying. I assume the runner is using --volume to bind/mount binary 'docker' into the runner w/c creates this unexpected empty docker directory when binary docker does not exist yet. Can this be to --mount as mentioned here changed https://github.com/moby/moby/issues/13121#issuecomment-336186993 so that this does not happen.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I'd like to inform you that I've created a support ticket as I need a a link for a build, but I can't ask for those links here due to privacy reasons.
I'll contact you through a support ticket shortly.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
I had to stop my runners and remove the runner folder completely , not just the "docker" sub-folder, which was wrongly mounted and shouldn't be a folder.
When I started the runners again, a "docker" executable is mounted instead.
I hope this helps.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.