"Connection reset by peer in connection to bitbucket.org:443" in Pipelines

[Rudolf Byker]

I'm using a self-hosted runner, version 1.369 (the latest at the time of writing). It was working well for a long time, but recently, without any infrastructure changes, I started getting this error on all steps:

We couldn't clone the repository. Try rerunning the pipeline.

In the "build setup", I see this output:

+ umask 000
+ GIT_LFS_SKIP_SMUDGE=1 retry 6 git clone --branch="..." https://x-token-auth:$REPOSITORY_OAUTH_ACCESS_TOKEN@bitbucket.org/$BITBUCKET_REPO_FULL_NAME.git $BUILD_DIR
Cloning into '/opt/atlassian/pipelines/agent/build'...
fatal: unable to access 'https://bitbucket.org/.../....git/': OpenSSL SSL_connect: Connection reset by peer in connection to bitbucket.org:443 
Attempt 1 of 6 failed.
Retrying in 1 seconds
Cloning into '/opt/atlassian/pipelines/agent/build'...
fatal: unable to access 'https://bitbucket.org/.../....git/': OpenSSL SSL_connect: Connection reset by peer in connection to bitbucket.org:443 
Attempt 2 of 6 failed.
Retrying in 2 seconds

etc...

The setup

This runner is in kubernetes on a K3s cluster. It's running on a development node which is a physical machine in my house. I manage this cluster, and nothing changed in the network.

The deployment for my runner instance is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: bbp-runner
  name: runner
  labels:
    app: bbp-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bbp-runner
  template:
    metadata:
      labels:
        app: bbp-runner
        accountUuid: foo
        runnerUuid: bar
    spec:
      nodeSelector:
        env: prod
      containers:
        - name: bitbucket-k8s-runner
          image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
          imagePullPolicy: Always
          env:
            - name: ACCOUNT_UUID
              value: "{foo}"
            - name: RUNNER_UUID
              value: "{bar}"
            - name: OAUTH_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: runner-oauth-credentials
                  key: oauthClientId
            - name: OAUTH_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: runner-oauth-credentials
                  key: oauthClientSecret
            - name: WORKING_DIRECTORY
              value: "/tmp"
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: docker-containers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: var-run
              mountPath: /var/run
        - name: docker-in-docker
          image: docker:20.10.7-dind
          securityContext:
            privileged: true
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: docker-containers
              mountPath: /var/lib/docker/containers
            - name: var-run
              mountPath: /var/run
      volumes:
        - name: tmp
          emptyDir: {}
        - name: docker-containers
          emptyDir: {}
        - name: var-run
          emptyDir: {}

The relevant containers are

docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner

and

docker:20.10.7-dind

I followed this guide https://janosmiko.com/blog/2021-09-08-bitbucket-pipelines-runners-in-k8s/ when I set it up a few months ago.

[Syahrul]

Hey @Rudolf Byker 

G'day.

The error below means the TLS handshake failed to establish the connection.

OpenSSL SSL_connect: Connection reset by peer in connection to bitbucket.org:443 "

This can happen for multiple reasons, but generally, it's caused by the local network.

I suggest checking with your network team to see if there's any recent change in your Firewall/proxy.

That said, you can also investigate this by running a few tests, such as:

1. Try cloning a repository from your Runner server with verbose logs enable to get more information

   GIT_TRACE_PACKET=1 GIT_TRACE=1 GIT_CURL_VERBOSE=1 <git command_here>
   

2. Run Curl to see if there's any issue with TLS connection from your server 

   curl -v https://bitbucket.org

If both tests failed with the same runner error, then most likely, the connection is being blocked on your Firewall/Proxy level.

I hope this helps.

Cheers,
Syahrul

[Rudolf Byker]

Thanks for your time.

I added some info about the infrastructure setup to the original post above.

I get a shell in the pod by doing this:

kubectl -n bbp-runner exec -it deploy/runner -- /bin/sh

The `docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner` image does not contain `git` or `curl`, but I could add it with `apk`, and then test what you suggested:

apk add curl git openssh

curl

curl -v https://bitbucket.org

This works fine. I get a 200 response with lots of HTML.

git

git clone git@bitbucket.org:rscdevelopers/living_word_vue.git

This works fine as well. I get the RSA key fingerprint, and then permission denied as expected, since I don't have the keys set up. This is enough to verify that TLS is working.

[Syahrul]

Hey @Rudolf Byker 

Thanks for the update.

We may need to investigate this further, so I went ahead and created a support ticket on your behalf. Please review it here.

One of our Support should get back to you within your timezone shortly.

Cheers,
Syahrul