As of yesterday at 6pm my time all self hosted bitbucket runners were working fine and we had our last successful build. This morning at 9:30 am all builds on self hosted runners are failing. I am struggling greatly trying to figure out what is happening. From what I can see.
Last successful build version details
Runner labels: self.hosted, linux, retail.science
latest: 1.555
First failed build version details
The version of this runner is outdated. Upgrade to the latest version (1.559).
So it seems a new version came out, but we were not using it yet. However the build had failed with the following error.
Service 'docker' exited with exit code: 1.
Here are the logs from the docker tab.
cat: can't open '/proc/net/arp_tables_names': No such file or directory
iptables v1.8.10 (nf_tables)
time="2024-02-02T16:32:14.749486743Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:14.749614343Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:15.749805000Z" level=warning msg="Binding to an IP address without --tlsverify is deprecated. Startup is intentionally being slowed down to show this message" host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:15.749876257Z" level=warning msg="Please consider generating tls certificates with client validation to prevent exposing unauthenticated root access to your network" host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:15.749903598Z" level=warning msg="You can override this by explicitly specifying '--tls=false' or '--tlsverify=false'" host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:15.749921619Z" level=warning msg="Support for listening on TCP without authentication or explicit intent to run without authentication will be removed in the next release" host="tcp://0.0.0.0:2375"
time="2024-02-02T16:32:30.800604163Z" level=warning msg="failed to load plugin io.containerd.internal.v1.opt" error="mkdir /opt/containerd: read-only file system"
time="2024-02-02T16:32:30.800906129Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2024-02-02T16:32:30.808994889Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
failed to start daemon: Error initializing network controller: error creating default "bridge" network: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: iptables --wait -t nat -I POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE: Warning: Extension MASQUERADE revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): CHAIN_ADD failed (No such file or directory): chain POSTROUTING
(exit status 4))
If I switch this to use a cloud hosted runner it all works except of course for the part of the build that is trying to access internal resources. I have tried so many different versions of the runner including the latest but nothing seems to be working and I get the same error message. Can anyone point me in the right direction here?
These runners are running on a Google Kubernetes cluster. We have three runners and all three started failing with the same error. I even tried to create an entirely new node pool for these runners but same error. GKE version is 1.29.0-gke.1381000.
I even tried moving it to a GKE cluster that is several versions older, but same error.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.