Bitbucket Runner stuck in Cloning

[Joar Buitrago]

I created a new Pipelines Runner Image based on Bitbucket Pipelines Runners Autoscaler.

The image is for usage with AWS ECS.

This image takes base from the official Bitbucket Pipelines Runner image, it only adds a wrapper that registers a new instance, and deregister it when the container is stopped.

"Removes" one env variable and adds three more, from the Bitbucket API.

Here is the Dockerfile

FROM docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:latest

RUN apt-get update && apt-get upgrade --assume-yes

RUN apt-get install --assume-yes jq

RUN sed -i 's/^java/exec java/gi' /opt/atlassian/pipelines/runner/entrypoint.sh

COPY entrypoint.sh /entrypoint.sh

ENTRYPOINT [ "/entrypoint.sh" ]

And the new Entrypoint

#!/usr/bin/env sh

set -e

printf "Welcome to Bitbucket Pipelines Runner Autoscaler\n\n"




access_token()
{
    printf "Getting Access Token...   " >&2

    if test -z $BITBUCKET_API_CLIENT_ID
    then
        printf "\n[ERROR] BITBUCKET_API_CLIENT_ID environment variable is not set.\n" >&2
        exit 1
    fi

    if test -z $BITBUCKET_API_CLIENT_SECRET
    then
        printf "\n[ERROR] BITBUCKET_API_CLIENT_SECRET environment variable is not set.\n" >&2
        exit 1
    fi

    echo -n $(curl \
        --fail \
        --silent \
        --user-agent 'Bitbucket-Pipelines-Runner-Autoscaler/1.0' \
        --request POST \
        --url https://bitbucket.org/site/oauth2/access_token \
        --header 'Authorization: Basic '$(printf "$BITBUCKET_API_CLIENT_ID:$BITBUCKET_API_CLIENT_SECRET" | basenc --wrap 0 --base64) \
        --header 'Content-Type: application/x-www-form-urlencoded' \
        --data grant_type=client_credentials | jq -r .access_token)

    printf "DONE\n" >&2
}





access_token_create=$(access_token)

printf "Registering new Pipelines Runner...   " >&2

runner_request=$(cat <<EOF
{
    "name": "AWS ECS {$(cat /proc/sys/kernel/random/uuid)}",
    "labels": [
        "self.hosted",
        "linux",
        "docker",
        "aws.ecs.managed"
    ]
}
EOF
)
runner_request=$(echo $runner_request | jq -c)

runner_info=$(curl \
    --fail \
    --silent \
    --user-agent 'Bitbucket-Pipelines-Runner-Autoscaler/1.0' \
    --request POST \
    --url 'https://api.bitbucket.org/internal/workspaces/'$BITBUCKET_WORKSPACE'/pipelines-config/runners' \
    --header 'Authorization: Bearer '$access_token_create \
    --header 'Content-Type: application/json' \
    --data "$runner_request")

export RUNNER_UUID=$(echo $runner_info | jq -r .uuid)
export OAUTH_CLIENT_ID=$(echo $runner_info | jq -r .oauth_client.id)
export OAUTH_CLIENT_SECRET=$(echo $runner_info | jq -r .oauth_client.secret)

echo $RUNNER_UUID > /opt/atlassian/pipelines/runner/uuid

unset access_token_create

printf "DONE\n" >&2

drain()
{
    trap - CHLD
    trap - KILL
    trap - TERM
    trap - INT

    if kill -0 $pid
    then
        kill -s TERM $pid
    fi
}

trap drain INT
trap drain TERM
trap drain KILL
trap drain CHLD





printf "\n\n\n\n\n" >&2
printf "Starting Pipelines Runner...\n\n" >&2
/opt/atlassian/pipelines/runner/entrypoint.sh &
pid=$!
wait $pid || true
wait $pid || true





printf "\n\n\n\n\n" >&2

access_token_delete=$(access_token)

printf "Deregistering Pipelines Runner...   " >&2

curl \
    --fail \
    --silent \
    --user-agent 'Bitbucket-Pipelines-Runner-Autoscaler/1.0' \
    --request DELETE \
    --url 'https://api.bitbucket.org/internal/workspaces/'$BITBUCKET_WORKSPACE'/pipelines-config/runners/'$(echo -n $RUNNER_UUID | jq -sRr @uri) \
    --header 'Authorization: Bearer '$access_token_delete

printf "DONE\n" >&2

The container is connected to a DinD image, for maximum isolation.

The containers have 2 volumes named:

• docker - mounted on /var/lib/docker
• tmp  - mounted on /tmp

In both containers, the docker volume is read-only for the runner container, the same as in documentation. This is mounted to /var/lib/docker and not /var/lib/docker/containers as DinD image defines a volume in /var/lib/docker, could be more specific and mount a folder inside the volume only too.

There is no docker.sock bind, but the DinD container exposes port 2375, and the runner is configured to send the requests to that port.

The runner starts correctly, recognizes the docker instance, reports the ONLINE status to Bitbucket.

After that, when the first job arrives, the runner is locked in the Cloning state and holds this state until timeout.

The /tmp/{RUNNER_UUID} seems to be filled with the files needed.

I do not know why the Runner locks in this state.

Probably related to DinD image, but I want to keep the full service isolated.

[Joar Buitrago · Accepted solution]

I just figure it out.

ECS Service Connect exposing port 2375 for Docker messes with the ability of Bitbucket Pipelines Runner to connect successfully if another service of docker is open within the step.

How do I solve it?

Create another volume mounting /var/run and docker container always needs to start first, that way, Bitbucket communicates with Docker via the socket and do not mess with the ports in the machine.