Runner process unable to pull images (500: no basic auth credentials) from private ECR repo

[Camilo Bernal]

I have a self-hosted runner configured on a Kubernetes cluster (EKS) and having some issues with the underlying runner process pulling images from ECR at runtime.

I've uploaded the auxiliary images from Use your Docker images in self-hosted runners | Bitbucket Cloud | Atlassian Support into a private repo.

There is no issues pulling images from the cluster or even from within the containers (I've manually logged into the appropiate ECR repo), but it looks like the runner has a java process that is triggered when a step is executed which pulls additional images at run time. This is where I get the following issue:

Status 500: {"message":"Head https://XXX.dkr.ecr.eu-west-1.amazonaws.com/v2/XXX/tools/manifests/prod-stable: no basic auth credentials"}

It seems odd that the Java process doesn't use the docker config in the container.

Below is the redacted manifest I'm using to deploy the runner (note that I can deploy this runner successfully, the issue appears at job execution):

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: runner-oauth-credentials
      labels:
        accountUuid: XXX
        repositoryUuid: XXX
        runnerUuid: XXXX
    data:
      oauthClientId: XXX=
      oauthClientSecret: XXX==
  - apiVersion: batch/v1
    kind: Job
    metadata:
      name: runner
    spec:
      template:
        metadata:
          labels:
            accountUuid: XXX
            repositoryUuid: XXX
            runnerUuid: XXX
        spec:
          containers:
            - name: runner
              image: XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/bitbucket/pipeline-runner:1.435
              resources:
                limits:
                  memory: "8Gi"
                  cpu: "2"
              env:
                - name: ACCOUNT_UUID
                  value: "{XXX}"
                - name: REPOSITORY_UUID
                  value: "{XXXX}"
                - name: RUNNER_UUID
                  value: "{XXX}"
                - name: OAUTH_CLIENT_ID
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientId
                - name: OAUTH_CLIENT_SECRET
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientSecret
                - name: WORKING_DIRECTORY
                  value: "/tmp"
                - name: PAUSE_IMAGE
                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/bitbucket/pipeline-pause:latest"
                - name: AUTH_PROXY_IMAGE
                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/pipeline/XXX/pipeline-auth-proxy:prod-stable"
                - name: CLONE_IMAGE
                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/pipeline-tools:prod-stable"
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                  readOnly: true # the runner only needs to read these files never write to them
                - name: var-run
                  mountPath: /var/run
            - name: docker-in-docker
              image: XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/docker:20.10.7-dind
              resources:
                limits:
                  memory: "8Gi"
                  cpu: "2"
              securityContext:
                privileged: true # required to allow docker in docker to run and assumes the namespace your applying this to has a pod security policy that allows privilege escalation
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                - name: var-run
                  mountPath: /var/run
          imagePullSecrets:
            - name: ecr-credentials
          restartPolicy: OnFailure # this allows the runner to restart locally if it was to crash
          volumes:
            - name: tmp # required to share a working directory between docker in docker and the runner
            - name: docker-containers # required to share the containers directory between docker in docker and the runner
            - name: var-run # required to share the docker socket between docker in docker and the runner
        # backoffLimit: 6 # this is the default and means it will retry upto 6 times if it crashes before it considers itself a failure with an exponential backoff between
        # completions: 1 # this is the default the job should ideally never complete as the runner never shuts down successfully
        # parallelism: 1 # this is the default their should only be one instance of this particular runner

[Mindaugas Z]

today i have same problem with self hosted runner ( yesterday everything worked ) 

also got error that runner is outdated. so upgraded them, and : 

Unable to pull image: Head \"https://715509311748.dkr.ecr.us-west-2.amazonaws.com/v2/sox/atlassian/pipelines-auth-proxy/manifests/prod-stable\": no basic auth credentials

[kevin_flammang]

This morning same problem for us (yesterday everything worked too)

We also upgraded the runner but it didn't help. 

[Mindaugas Z]

still happening  https://bitbucket.status.atlassian.com/incidents/19b4rq4rwfqw