Self-hosted runners failing with timeout

[Joris Vleminckx]

Hello,

Our self-hosted runners have started to fail with the below error:

Unable to pull image: Head \"https://docker-public.packages.atlassian.com/v2/sox/atlassian/bitbucket-pipelines-docker-daemon/manifests/v20.10.24-multiarch-prod-stable\": Get \"https://docker-public.packages.atlassian.com/api/docker/docker-public/v2/token?scope=repository%3Asox%2Fatlassian%2Fbitbucket-pipelines-docker-daemon%3Apull&service=docker-public.packages.atlassian.com\": dial tcp 65.9.121.45:443: i/o timeout (Client.Timeout exceeded while awaiting headers)

When I do a manual curl from inside the runner, I obtain an UNAUTHORIZED error:

curl -v https://docker-public.packages.atlassian.com/v2/
* Trying 13.227.200.77:443...
* TCP_NODELAY set
* Connected to docker-public.packages.atlassian.com (13.227.200.77) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=packages.atlassian.com
* start date: May 12 00:00:00 2023 GMT
* expire date: Jun 10 23:59:59 2024 GMT
* subjectAltName: host "docker-public.packages.atlassian.com" matched cert's "*.packages.atlassian.com"
* issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56552152e8c0)
> GET /v2/ HTTP/2
> Host: docker-public.packages.atlassian.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 401 
< content-type: application/json;charset=ISO-8859-1
< content-length: 87
< date: Thu, 20 Jul 2023 13:43:13 GMT
< server: AtlassianEdge
< docker-distribution-api-version: registry/2.0
< www-authenticate: Bearer realm="https://docker-public.packages.atlassian.com/api/docker/docker-public/v2/token",service="docker-public.packages.atlassian.com"
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< atl-traceid: b6212e2db47912cd
< report-to: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
< nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
< strict-transport-security: max-age=63072000; preload
< x-cache: Error from cloudfront
< via: 1.1 10c107d4632ead14a9348b11e8214210.cloudfront.net (CloudFront)
< x-amz-cf-pop: SCL50-C1
< x-amz-cf-id: _HxIycys5riFuhLDHrNb-PVhKCEYH-WIrYbmVJdvQgfoYOpgC6TMQA==
< 
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host docker-public.packages.atlassian.com left intact

On https://bitbucket.status.atlassian.com/ I see everything green.

This issue looks similar to https://community.atlassian.com/t5/Bitbucket-questions/Docker-Public-Packages-down/qaq-p/2218434

Could anyone shed any light on this issue? This is breaking all our pipelines...

[Patrik S]

Hello @Joris Vleminckx and welcome to the Community!

We are not currently tracking any incident related to Pipeline's runner's image. From the error message being returned, it seems like a timeout in the connection. I would suggest checking if there were no recent changes in the runner's host network that might be blocking the connection.

Also, if your runner is not on the latest version, you can try updating it by running the following commands in the host machine :

1. Pull the newest runner's images :

   docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
   

2. Remove the existing runner container(s) :

   docker container rm  runner-12b345e6-b789-1e2b-3da4-5cda67c8ff9c
   

   Where 12b345e6-b789-1e2b-3da4-5cda67c8ff9c is the UUID of your runners. If you don't remember the container name, you can list all the containers in the host machine using the command :

   docker container ls
   

3. Start the runner container(s) again with the pre-defined command you were given when you created the runner. In case you don't have the command saved, you can create a new runner in the UI and you will be given a new command.

With the runners updated, you can try to trigger the pipeline again and check if the error is fixed.

Hope that helps! Let me know in case you have any questions.

Thank you, @Joris Vleminckx !

Patrik S

[Joris Vleminckx]

Hi Patrik,

We regularly update our runners to the latest version, so we are already running the latest version. :-)

I think the timeout message from the pipeline is a red herring, because the manual `curl` clearly indicates a strange "UNAUTHORIZED" message against the bitbucket servers.

I've recently checked and today our pipelines seem to be running normally again.