Selfhosted Linux Docker Runner failing with "No such container: ..._system_auth-proxy"

Chris Flora

When running my pipeline against my freshly spun up docker installation, I'm getting this error before any of the scripts run:

Status 404: {"message":"No such container: 67d5e5b0-717b-56f7-bcb9-ffdc1e351cf2_f6ff8605-0455-443a-b606-394a84893155_system_auth-proxy"}

I don't see anything of note in the docker logs - I see the "Remove" logged, and then immediately the "Creating" is logged, and then two seconds later it "Inspects" and fails.  This happens immediately after it "Inspects" my primary image.

I haven't been able to find anything about this "system_auth-proxy" image, and have no idea why it's failing.  Any ideas?  The exact same config works fine with the atlassian runners.

I have docker running on a clean Ubuntu 22.04 VM, and used the docker documentation to install from their repos.  CPU, memory, and diskspace are nowhere near the limits for the VM.  I also tried installing docker on bare metal, and see the same results.

Here's the full bitbucket-pipeline.yaml if it helps.  This is getting pushed to the develop branch. I've tried installing hugo from scratch in the build steps instead of using a prebuilt image with no difference.

Any help or ideas on where to look are appreciated!!

pipelines:
  default:
    - step:
        name: Default no-op
        script:
          - echo "Only pushing develop and main branches to Firebase Hosting"
  branches:
    main:
      - step:
          name: Build
          runs-on:
            - self.hosted
            - linux
          image: hugomods/hugo:0.122.0
          script:
            - hugo --cleanDestinationDir --minify -e production
          artifacts:
            - public/**
            - step:
                name: Deploy to Firebase Hosting
                runs-on:
                  - self.hosted
                  - linux
                deployment: production
                script:
                  - pipe: atlassian/firebase-deploy:5.1.0
                    variables:
                      FIREBASE_TOKEN: $FIREBASE_TOKEN
                      PROJECT_ID: $FIREBASE_PROJECT
                      EXTRA_ARGS: --only hosting:$FIREBASE_SITE
    develop:
      - step:
          name: Build the Hugo site
          runs-on:
            - self.hosted
            - linux
          image: hugomods/hugo:0.122.0
          script:
            - hugo -DEF --cleanDestinationDir --minify -e develop
          artifacts:
            - public/**
       - step:
           name: Deploy to Firebase Hosting
           runs-on:
             - self.hosted
             - linux
           deployment: develop
           image: node:18.16.1
           caches: 
             - node
           script:
             - npm install -g firebase-tools:13.1.0
             - firebase deploy --token $FIREBASE_TOKEN --project $FIREBASE_PROJECT --only hosting:$FIREBASE_SITE

 

Chris Flora

The problem appears to have been something on my Proxmox server.  My original runner was set up in a Proxmox VM running Ubuntu 22.04.  I then tried a second time with docker installed directly on the Proxmox server - received the same error in both cases.  Since my testing had me leaning towards a problem on the docker side instead of the bitbucket side, and the only lead I found in my searching with a similar issue pointed to a problem with SELinux (which isn't installed on Proxmox, but it made me think maybe there's something with AppArmor or some root Proxmox dependency), I decided to try spinning up a new bare-metal Ubuntu server on my network.  And bam, worked immediately.

So for anyone digging this thread up in the future, if you're running on Proxmox, try running it outside of Proxmox.  I am not planning on digging any deeper here since I was already planning on moving away from Proxmox for other reasons.  But good luck!