Self-hosted runner: step set up fails when trying to use docker service. ExitCode: Some(4)

[Scott Crunkleton]

I'm running a self-hosted runner and whenever I try to enable docker support for individual steps, or for all steps via `options: docker: true`, the pipeline fails in the step set up. The pipeline dashboard doesn't include any helpful error logs. If I log into my runner and check the stdout, there is a always line `[2021-07-31 07:40:08,139] Container has state (exitCode: Some(4), OOMKilled Some(false))`.

Here is my bitbucket-pipelines.yml:

image:
  name: gcr.io/novo00/bitbucket-pipelines:latest
  username: _json_key
  password: '$CREDS'
options:
  docker: true
pipelines:
  branches:
    master:
      - step:
          runs-on:
            - self.hosted
          script:
            - echo "Starting script"
            - build/bitbucket_pipelines/version.sh
      - step:
          runs-on:
            - self.hosted
          script:
            - echo "Starting script"
            - build/bitbucket_pipelines/build.sh
      - step:
          runs-on:
            - self.hosted
          script:
            - echo "Starting script"
            - build/bitbucket_pipelines/deploy_no_traffic.sh

This fails during set up of step 1. If I remove `options: docker: true` and put it as a service to step 2 (where it is actually needed), step 1 runs fine, and the pipeline fails during set up of step 2.

As instructed, I have the docker socket and containers dir mounted properly: `-v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/containers:/var/lib/docker/containers:ro`

Is this an issue? Or am I doing something wrong?

EDIT: This pipeline works in Bitbucket's runners. It breaks when self-hosted.

[Sean Reifschneider]

I'm also running into this, also using the docker service and seeing "Container has state (exitCode: Some(1), OOMKilled Some(false))".  I tried doing "size: 2x" as well, but that didn't help.

[Sean Reifschneider]

FYI: I've created a support ticket for this issue: BBS-171239

[f.hess]

When your ticket is done, can you perhaps log the status over here? I'd love to hear the outcome as we run into the same issue :) 

[Sean Reifschneider]

Resolved, see my other top level comment starting "RESOLVED:".

[Karthik]

We are facing same issue all of a sudden. It was working and started failing. We are currently running 1.555 version and looks like there is a new version which is 1.559 but no luck still same issue. 

[Elias Balasis]

I think I am experiencing the same

Container has state (exitCode: Some(0), OOMKilled Some(false))

see https://community.atlassian.com/t5/Bitbucket-questions/Workspace-Runner-Errors/qaq-p/2003147#M80852

[Artem Gnatuk]

Maybe my solution wil be helpfull for somebody. So...

All work fine when CPU have L1 cache <= 32k. When L1 cahe bigger container system_auth_proxy crashed at start with error:

Container has state (exitCode: Some(1), OOMKilled Some(false))

 Because nginx option server_names_hash dafault value must be adjusted according cache size.

[Tim Kovalev]

Does it still fail in the latest runner?

I opened an issue a couple of months ago, and atlassian fixed it, so it should be working now.

[Artem Gnatuk]

Yes. It fail with all version of runner.

[Sean Reifschneider]

I also had this problem months ago, opened a case, and they got it fixed with a new runner.  Has been running great since then.  See my other comments in this thread from Aug 21.

[Sean Reifschneider]

RESOLVED: Atlassian pushed a new docker image yesterday which has a fix for this issue for me.  It was apparently an issue with the nginx component in the docker image setting a bucket size too small and causing problem.  I deleted my bitbucket runner container, deleted the cached image (doing a pull wasn't updating it), and then pulled it again and launched it.  Working great now!  Thanks Atlassian Support!

[Joseph Scheidt]

What tag did you pull to get this new build? When I pull tag `1` and `latest` and `1.184` I get an image built 3 weeks ago (~August 26th 2021). Doesn't seem to contain the fix, as I am still having this issue.

[Fabian Pötsch]

Tried with following tags: 1, 1.171, 1.184, latest, stg-stable, stg-beta, prod-beta, prod-stable. None of them work.

What no one seems to have posted, is this error log from the docker daemon:

level=warning msg="reference for unknown type: application/vnd.docker.distribution.manifest.v1+prettyjws" digest="sha256:e8fc56926ac3d5705772f13be fbaee3aa2fc6e9c52faee3d96b26612cd77556c" remote="docker-hub.packages.atlassian.com/google/pause:latest"
level=error msg="(*service).Write failed" error="rpc error: code = FailedPrecondition desc = unexpected commit digest sha256:cbb11e46703e62688f97c 6f12f9c02e7a6568f572cd36042d677a46b2a40b5c6, expected sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c: failed precondition" expected="sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c"ref="unknown-sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c"
level=warning msg="Error persisting manifest" digest="sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c" error="error committing manifest to content store: commit failed: unexpected commit digest sha256:cbb11e46703e62688f97c6f12f9c02e7a6568f572cd36042d677a46b2a40b5c6, expected sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c: fai led precondition" remote="docker-hub.packages.atlassian.com/google/pause:latest"

The calculated sha256 for the google/pause image hosted by BitBucket does not match.

 I tried clearing all caches or other data related to docker. Nothing changes this behavior.

[feandoe]

I still have the same error even with the last one (1.240)

[Artem Gnatuk]

Did you use deb-based distro?

[feandoe]

yes, Ubuntu 20.04.3 LTS

[Artem Gnatuk]

Try CentOS 7  - it help me

[Joseph Scheidt]

I can confirm that this problem is related to the OS. Specifically it is a filesystem problem. We were told by Atlassian support that the filesystem must support 'd_type' (see output of docker system info).

We saw this issue with Debian 11, however Debian 10 and the latest Ubuntu LTS seem to support the appropriate filesystem to run docker-in-docker for the self-hosted runners.

[feandoe]

Thanks to both. On Ubuntu 20 LTS it didn't work, but we try on Ubuntu 18 and it worked!

[Joseph Scheidt]

Great! I'll speak with our sysadmin. He might have made changes to the Ubuntu 20 LTS filesystem when creating the VM, if you were not able to make it work off the shelf.

[Sean Reifschneider]

FYI: I have my runner working on 20.04.  I installed docker from the docker.com repo, and "docker system info" reports "Supports d_type: true"    Root fs is ext4.

[Tim Kovalev]

I suggest you to open a ticket in atlassian support, if you haven't yet. It helped me - they started to work on the issue and fixed it for my specific case.

[Alexander Kaiser]

Are there any new insights to this problem?

I experience the same issue on Ubuntu 20.04 with runner version 1.248 (EDIT: also with 1.250 which was released today) .

I observe a similar error message as @Fabian Pötsch reported in the deamon logs whenever I start a pipeline.

Its hard to track down the problem because the containers seem to be deleted before I could check any error messages within spawned the containers.

---

Deamon log error messages (journalctl -fu docker.service):

"reference for unknown type: application/vnd.docker.distribution.manifest.v1+prettyjws" digest="sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c" remote="docker-hub.packages.atlassian.com/google/pause:latest"

"Error persisting manifest" digest="sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c" error="error committing manifest to content store: commit failed: unexpected commit digest sha256:a48610965f03ff63a1be7cf8912b46112268f8b6ec7912f6f6a42d7cece41d90, expected sha256:e8fc56926ac3d5705772f13befbaee3aa2fc6e9c52faee3d96b26612cd77556c: failed precondition" remote="docker-hub.packages.atlassian.com/google/pause:latest"

Thanks in advance,

Alex

[Jannick Gröpper]

I had the same problem. The runner stopped working since we have upgraded to Debian 11. Now we have deployed the runner on a Debian 10 server and it works. 

[tilot]

As I don't have access to support raising page either, I post my docker system info:

docker system info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.8.0)

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 6
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.15.0
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 32GiB
Name: h2931691.stratoserver.net
ID: DZET:P36T:ZNRG:R7NN:SQC4:FEMK:VYDP:URTO:YJ7Y:SIKX:OVQI:SDKU
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

runner.log

[2021-08-10 13:20:30,343] Setting runner state to executing step.
[2021-08-10 13:20:30,345] Getting step StepId{accountUuid={cabfe88d-8729-4fdf-933e-c3f3dfe42c4b}, repositoryUuid={442cf95a-bc37-461b-94d6-bc3a452aaef5}, pipelineUuid={6d50c301-817d-49f9-b8e7-351b234270b7}, stepUuid={aa071d02-ad49-465f-a45e-a66280795484}}.
[2021-08-10 13:20:30,347] Getting oauth token for step.
[2021-08-10 13:20:30,349] Getting environment variables for step.
[2021-08-10 13:20:30,740] Getting all artifacts for step.
[2021-08-10 13:20:30,743] Getting SSH private key.
[2021-08-10 13:20:30,750] Getting known hosts.
[2021-08-10 13:20:30,969] SSH private key not found
[2021-08-10 13:20:30,976] Setting up directories.
[2021-08-10 13:20:30,978] Starting log uploader.
[2021-08-10 13:20:30,982] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_clone
[2021-08-10 13:20:30,993] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_build
[2021-08-10 13:20:30,996] Setting up step timeout: PT2H
[2021-08-10 13:20:30,997] Starting websocket listening to STEP_COMPLETED events.
[2021-08-10 13:20:30,998] Checking for step completion every PT30S seconds.
[2021-08-10 13:20:31,225] Updating step progress to PULLING_IMAGES.
[2021-08-10 13:20:31,446] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable.
[2021-08-10 13:20:31,983] Appending log line to main log.
[2021-08-10 13:20:32,006] Pulling image node:12.22.3.
[2021-08-10 13:20:33,430] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable.
[2021-08-10 13:20:33,934] Pulling image google/pause:latest.
[2021-08-10 13:20:34,528] Updating runner state to "ONLINE".
[2021-08-10 13:20:35,811] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_pause
[2021-08-10 13:20:35,816] Updating step progress to CLONING.
[2021-08-10 13:20:35,823] Creating container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_pause.
[2021-08-10 13:20:35,887] Starting container.
[2021-08-10 13:20:36,070] Generating clone script.
[2021-08-10 13:20:36,075] Creating container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_clone.
[2021-08-10 13:20:36,075] Executing clone script in clone container.
[2021-08-10 13:20:36,328] Starting container.
[2021-08-10 13:20:36,539] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_system_auth-proxy
[2021-08-10 13:20:36,544] Creating container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_system_auth-proxy.
[2021-08-10 13:20:36,778] Starting container.
[2021-08-10 13:20:36,914] Adding container log: /var/lib/docker/containers/9dc6cd5dc7288b09552ae4ccfc7d8da4b31a5987047ce8b1def1d5ef6c9b6978/9dc6cd5dc7288b09552ae4ccfc7d8da4b31a5987047ce8b1def1d5ef6c9b6978-json.log
[2021-08-10 13:20:36,915] Waiting on container to exit.
[2021-08-10 13:20:36,916] Creating exec into container.
[2021-08-10 13:20:36,921] Starting exec into container and waiting for exec to exit.
[2021-08-10 13:20:37,093] Adding container log: /var/lib/docker/containers/2a4b93f8106820d724e7261d6835f80198738d04f78a2f67742d8ef46e18acf4/2a4b93f8106820d724e7261d6835f80198738d04f78a2f67742d8ef46e18acf4-json.log
[2021-08-10 13:20:37,094] Waiting on container to exit.
[2021-08-10 13:20:37,599] Container has state (exitCode: Some(1), OOMKilled Some(false))
[2021-08-10 13:20:37,601] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_build
[2021-08-10 13:20:37,605] Not uploading caches. (numberOfCaches: 0, resultOrError: FAILED)
[2021-08-10 13:20:37,606] Not uploading artifacts. (numberOfArtifacts: 0, resultOrError: FAILED)
[2021-08-10 13:20:37,606] Updating step progress to PARSING_TEST_RESULTS.
[2021-08-10 13:20:37,845] Test report processing complete.
[2021-08-10 13:20:37,845] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_clone
[2021-08-10 13:20:37,986] Appending log line to main log.
[2021-08-10 13:20:37,986] Appending log line to log: {1312fe91-f992-417c-81c9-4ea3d126ad77}.
[2021-08-10 13:20:38,290] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_clone
[2021-08-10 13:20:38,294] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_build
[2021-08-10 13:20:38,297] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_system_auth-proxy
[2021-08-10 13:20:38,306] Removing container 841db0a0-3048-54e3-a45a-c100d5398793_aa071d02-ad49-465f-a45e-a66280795484_pause
[2021-08-10 13:20:38,724] Updating step progress to COMPLETING_LOGS.
[2021-08-10 13:20:38,972] Shutting down log uploader.
[2021-08-10 13:20:38,974] Tearing down directories.
[2021-08-10 13:20:38,982] Cancelling timeout
[2021-08-10 13:20:38,983] Completing step with result Result{status=FAILED, error=None}.
[2021-08-10 13:20:39,254] Setting runner state to not executing step.
[2021-08-10 13:20:39,254] Waiting for next step.

[Tim Kovalev]

Same problem with the latest runner:

`docker system info`:

Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.8.0)

Server:
Containers: 8
Running: 1
Paused: 0
Stopped: 7
Images: 7
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version: v1.0.1-0-g4144b63
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.66GiB
Name: 265441.simplecloud.ru
ID: 74JG:SHOY:BFJH:CTL4:LZJC:WQU4:QC4R:EFNX:JUWV:RSXP:I3TO:BQUG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

`runner.log`

[2021-08-03 11:57:09,686] Runner version: 1.167
[2021-08-03 11:57:09,740] Runner runtime: linux-docker
[2021-08-03 11:57:13,410] Copying Docker cli to working directory.
[2021-08-03 11:57:13,944] Starting websocket listening to RUNNER_UPDATED events.
[2021-08-03 11:57:14,156] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.
[2021-08-03 11:57:14,799] Updating runner state to "ONLINE".
[2021-08-03 11:57:16,801] Setting runner state to executing step.
[2021-08-03 11:57:16,864] Getting step StepId{accountUuid={5f96beb2-a182-4102-a152-0c9ff318b535}, repositoryUuid={82897585-e596-452f-99a8-0b388cd88a65}, pipelineUuid={e78ac544-2ec1-4f3b-9a6c-bbdbff725598}, stepUuid={3fa41c4c-225b-4f02-9d22-330bf6c12269}}.
[2021-08-03 11:57:16,893] Getting oauth token for step.
[2021-08-03 11:57:16,906] Getting environment variables for step.
[2021-08-03 11:57:17,668] Getting all artifacts for step.
[2021-08-03 11:57:17,690] Getting ssh private key.
[2021-08-03 11:57:17,701] Getting known hosts.
[2021-08-03 11:57:17,933] Translating HTTP 404 NOT_FOUND response into a com.atlassian.pipelines.stargate.client.core.exceptions.StargateNotFoundException
[2021-08-03 11:57:18,134] Setting up directories.
[2021-08-03 11:57:18,142] Starting log uploader.
[2021-08-03 11:57:18,174] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_clone
[2021-08-03 11:57:18,229] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_build
[2021-08-03 11:57:18,269] Setting up step timeout: PT2H
[2021-08-03 11:57:18,272] Starting websocket listening to STEP_COMPLETED events.
[2021-08-03 11:57:18,275] Checking for step completion every PT30S seconds.
[2021-08-03 11:57:18,554] Updating step progress to PULLING_IMAGES.
[2021-08-03 11:57:18,824] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable.
[2021-08-03 11:57:19,190] Appending log line to main log.
[2021-08-03 11:57:19,716] Pulling image atlassian/default-image:latest.
[2021-08-03 11:57:21,246] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable.
[2021-08-03 11:57:21,991] Pulling image google/pause:latest.
[2021-08-03 11:57:24,155] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_pause
[2021-08-03 11:57:24,173] Updating step progress to CLONING.
[2021-08-03 11:57:24,174] Creating container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_pause.
[2021-08-03 11:57:24,351] Starting container.
[2021-08-03 11:57:24,427] Generating clone script.
[2021-08-03 11:57:24,468] Creating container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_clone.
[2021-08-03 11:57:24,468] Executing clone script in clone container.
[2021-08-03 11:57:24,559] Starting container.
[2021-08-03 11:57:25,063] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_system_auth-proxy
[2021-08-03 11:57:25,069] Creating container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_system_auth-proxy.
[2021-08-03 11:57:25,164] Starting container.
[2021-08-03 11:57:25,283] Adding container log: /var/lib/docker/containers/7e3b5aafafea6277f7636df6ba438d6e12ce719538c41c176079c61ece1f3d6d/7e3b5aafafea6277f7636df6ba438d6e12ce719538c41c176079c61ece1f3d6d-json.log
[2021-08-03 11:57:25,300] Waiting on container to exit.
[2021-08-03 11:57:25,310] Creating exec into container.
[2021-08-03 11:57:25,331] Starting exec into container and waiting for exec to exit.
[2021-08-03 11:57:25,494] Adding container log: /var/lib/docker/containers/bd014162296a0990ea8e27a9c9eb7860fdbe292a2a142bca5bb39697d3362425/bd014162296a0990ea8e27a9c9eb7860fdbe292a2a142bca5bb39697d3362425-json.log
[2021-08-03 11:57:25,496] Waiting on container to exit.
[2021-08-03 11:57:25,929] Container has state (exitCode: Some(1), OOMKilled Some(false))
[2021-08-03 11:57:25,952] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_build
[2021-08-03 11:57:25,962] Not uploading caches. (numberOfCaches: 0, resultOrError: FAILED)
[2021-08-03 11:57:25,969] Not uploading artifacts. (numberOfArtifacts: 0, resultOrError: FAILED)
[2021-08-03 11:57:25,970] Updating step progress to PARSING_TEST_RESULTS.
[2021-08-03 11:57:26,183] Appending log line to log: {46dd0db7-59fe-4e46-8f5d-5e52a9ba50ec}.
[2021-08-03 11:57:26,188] Appending log line to main log.
[2021-08-03 11:57:26,724] Test report processing complete.
[2021-08-03 11:57:26,725] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_clone
[2021-08-03 11:57:27,007] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_clone
[2021-08-03 11:57:27,013] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_build
[2021-08-03 11:57:27,021] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_system_auth-proxy
[2021-08-03 11:57:27,036] Removing container 88deb90f-8b02-5918-97ed-c07759890d04_3fa41c4c-225b-4f02-9d22-330bf6c12269_pause
[2021-08-03 11:57:27,172] Appending log line to main log.
[2021-08-03 11:57:27,331] Updating step progress to COMPLETING_LOGS.
[2021-08-03 11:57:27,581] Shutting down log uploader.
[2021-08-03 11:57:27,587] Tearing down directories.
[2021-08-03 11:57:27,602] Cancelling timeout
[2021-08-03 11:57:27,614] Completing step with result Result{status=FAILED, error=None}.
[2021-08-03 11:57:27,884] Setting runner state to not executing step.
[2021-08-03 11:57:27,885] Waiting for next step. 

[Tim Kovalev]

The problem could be with the system_auth-proxy image. It crashes with nginx error log:
```
{"log":"2021/08/03 14:45:05 [emerg] 1#1: could not build server_names_hash, you should increase server_names_hash_bucket_size: 32\n","stream":"stderr","time":"2021-08-03T14:45:06.192372874Z"}
{"log":"nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 32\n","stream":"stderr","time":"2021-08-03T14:45:06.192459403Z"}
```

Probably you might have to change nginx settings, as described here:
https://stackoverflow.com/questions/13895933/nginx-emerg-could-not-build-the-server-names-hash-you-should-increase-server

[tilot]

Thanks for your answer. I tried this out with no success.

I changed the nginx on the host. I suppose, there is no nginx between runner and bitbucket caller. An I didn't find an nginx in my runner container instance.

[lassian]

Hi,

Can you please a raise a support case at https://support.atlassian.com/bitbucket-cloud/

That way we can investigate with you the error better.

Kind Regards,
Nathan Burrell

[Scott Crunkleton]

I try to start a support case via https://support.atlassian.com/contact/#/ but when I get to the end of the form, it only gives me two options:

Getting help from the Atlassian Community

Ask questions or browse discussions to find the information you need. Support staff and developers from Atlassian check in regularly to help answer tricky or highly technical questions.

Explore Community

Check out our Bitbucket how-to guides

You'll find info on how to set up Bitbucket and make the most of its features in our online documentation.

View Documentation

The options I used were "Technical issues and bugs", "Bitbucket", "bitbucket.org/novohealth", and "No"

[lassian]

Hi,

Ahh yep your on the free tier of usage, I dont believe that can raise support cases.

Can you please provide us than the output of

`docker system info` and also your complete runner log file which is located under your runners working directory (by default /tmp) its called runner.log.

That way we can better investigate this.

Kind Regards,

Nathan Burrell

[Scott Crunkleton]

`docker system info`

Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 7
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 69107e47a62e1d690afa2b9b1d43f8ece3ff4483.m
runc version: v1.0.1-0-g4144b638
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.13.5-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.768GiB
Name: novobbrunner
ID: SFOL:YTHG:LEQO:GVJI:L2SI:KB2M:G34B:ECA4:SVVK:7DYV:HLGV:RN3C
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

runner.log

[2021-08-03 03:04:33,371] Runner starting version: 1.161
[2021-08-03 03:04:33,425] Runner runtime: linux-docker
[2021-08-03 03:04:36,943] Copying Docker cli to working directory.
[2021-08-03 03:04:37,349] Starting websocket listening to RUNNER_UPDATED events.
[2021-08-03 03:04:37,600] Updating runner status to "ONLINE" and checking for new steps assigned to the runner after 0 seconds and then every 30 seconds.
[2021-08-03 03:04:38,166] Updating runner state to "ONLINE".
[2021-08-03 03:05:07,606] Updating runner state to "ONLINE".
[2021-08-03 03:05:37,606] Updating runner state to "ONLINE".
[2021-08-03 03:06:07,605] Updating runner state to "ONLINE".
[2021-08-03 03:06:25,722] Setting runner state to executing step.
[2021-08-03 03:06:25,749] Getting step StepId{accountUuid={bcc59374-462c-4a8d-86e1-1f3046617126}, repositoryUuid={9dc4490e-e14b-4b57-9552-621443325886}, pipelineUuid={4f3557b6-99da-40b5-8f60-84d28fd74842}, stepUuid={e2181c4d-243d-4156-994b-6bdbee2802f6}}.
[2021-08-03 03:06:25,770] Getting oauth token for step.
[2021-08-03 03:06:25,785] Getting environment variables for step.
[2021-08-03 03:06:26,472] Getting all artifacts for step.
[2021-08-03 03:06:26,506] Getting ssh private key.
[2021-08-03 03:06:26,511] Getting known hosts.
[2021-08-03 03:06:26,641] Translating HTTP 404 NOT_FOUND response into a com.atlassian.pipelines.stargate.client.core.exceptions.StargateNotFoundException
[2021-08-03 03:06:26,795] Setting up directories.
[2021-08-03 03:06:26,798] Starting log uploader.
[2021-08-03 03:06:26,840] Removing container clone
[2021-08-03 03:06:26,858] Removing container build
[2021-08-03 03:06:26,876] Setting up step timeout: PT2H
[2021-08-03 03:06:26,878] Starting websocket listening to STEP_COMPLETED events.
[2021-08-03 03:06:26,882] Checking for step completion every PT30S seconds.
[2021-08-03 03:06:27,059] Updating step progress to PULLING_IMAGES.
[2021-08-03 03:06:27,229] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-dvcs-tools:prod-stable.
[2021-08-03 03:06:27,698] Pulling image gcr.io/novo00/bitbucket-pipelines:latest.
[2021-08-03 03:06:27,836] Appending log line to main log.
[2021-08-03 03:06:27,935] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable.
[2021-08-03 03:06:28,380] Pulling image docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20-prod-stable.
[2021-08-03 03:06:28,806] Pulling image google/pause:latest.
[2021-08-03 03:06:29,506] Removing container pause
[2021-08-03 03:06:29,515] Updating step progress to CLONING.
[2021-08-03 03:06:29,521] Creating container pause.
[2021-08-03 03:06:29,617] Starting container.
[2021-08-03 03:06:29,675] Generating clone script.
[2021-08-03 03:06:29,710] Creating container clone.
[2021-08-03 03:06:29,711] Executing clone script in clone container.
[2021-08-03 03:06:29,789] Starting container.
[2021-08-03 03:06:30,107] Removing container system-auth-proxy
[2021-08-03 03:06:30,116] Creating container system-auth-proxy.
[2021-08-03 03:06:30,123] Removing container system-docker
[2021-08-03 03:06:30,138] Creating container system-docker.
[2021-08-03 03:06:30,184] Starting container.
[2021-08-03 03:06:30,225] Starting container.
[2021-08-03 03:06:30,296] Adding container log: /var/lib/docker/containers/bba726bd91489c5b739ced720b3bf08e616f35ac8ded0a4e6fdde3db1c91a743/bba726bd91489c5b739ced720b3bf08e616f35ac8ded0a4e6fdde3db1c91a743-json.log
[2021-08-03 03:06:30,304] Waiting on container to exit.
[2021-08-03 03:06:30,309] Creating exec into container.
[2021-08-03 03:06:30,334] Starting exec into container and waiting for exec to exit.
[2021-08-03 03:06:30,424] Adding container log: /var/lib/docker/containers/3fe321104d975c75cd3168fa94ad9f1096c29aaad0bfa2a141faf25c626d9d3a/3fe321104d975c75cd3168fa94ad9f1096c29aaad0bfa2a141faf25c626d9d3a-json.log
[2021-08-03 03:06:30,425] Waiting on container to exit.
[2021-08-03 03:06:30,529] Adding container log: /var/lib/docker/containers/cf4dc61911445cb322d76ff33c96326d8b9aaa3c7cf86086
1b4a4517f526e97f/cf4dc61911445cb322d76ff33c96326d8b9aaa3c7cf860861b4a4517f526e97f-json.log
[2021-08-03 03:06:30,530] Waiting on container to exit.
[2021-08-03 03:06:30,838] Appending log line to log: {0334e672-9a73-4b50-804d-c2e823219e16}.
[2021-08-03 03:06:30,865] Container has state (exitCode: Some(4), OOMKilled Some(false))
[2021-08-03 03:06:30,866] Appending log line to main log.
[2021-08-03 03:06:30,892] Removing container build
[2021-08-03 03:06:30,903] Not uploading caches. (numberOfCaches: 0, resultOrError: FAILED)
[2021-08-03 03:06:30,905] Not uploading artifacts. (numberOfArtifacts: 0, resultOrError: FAILED)
[2021-08-03 03:06:30,912] Updating step progress to PARSING_TEST_RESULTS.
[2021-08-03 03:06:31,086] Test report processing complete.
[2021-08-03 03:06:31,087] Removing container clone
[2021-08-03 03:06:31,100] Appending log line to log: {cde9fe19-4bbc-48e9-99b8-717a5c8b5fbf}.
[2021-08-03 03:06:31,244] Removing container clone
[2021-08-03 03:06:31,249] Removing container build
[2021-08-03 03:06:31,253] Removing container system-auth-proxy
[2021-08-03 03:06:31,375] Removing container system-docker
[2021-08-03 03:06:31,391] Removing container pause
[2021-08-03 03:06:31,571] Updating step progress to COMPLETING_LOGS.
[2021-08-03 03:06:31,722] Shutting down log uploader.
[2021-08-03 03:06:31,726] Appending log line to main log.
[2021-08-03 03:06:31,969] Tearing down directories.
[2021-08-03 03:06:31,973] Cancelling timeout
[2021-08-03 03:06:31,978] Completing step with result Result{status=FAILED, error=None}.
[2021-08-03 03:06:32,137] Setting runner state to not executing step.
[2021-08-03 03:06:32,138] Waiting for next step.
[2021-08-03 03:06:37,606] Updating runner state to "ONLINE".
[2021-08-03 03:07:07,606] Updating runner state to "ONLINE".

[Artem Gnatuk]

and show lscpu too

[tilot]

It's the same with me

It breaks only in a self hosted runner:

tilo@h2931691:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
tilo@h2931691:~$ docker -v
Docker version 20.10.7, build f0df350

[2021-07-31 16:56:27,168] Executing clone script in clone container.
[2021-07-31 16:56:27,281] Starting container.
[2021-07-31 16:56:27,532] Removing container system-auth-proxy
[2021-07-31 16:56:27,542] Creating container system-auth-proxy.
[2021-07-31 16:56:27,645] Adding container log: /var/lib/docker/containers/66b2ecc6497c13806dedf5f57a7303442159ace464b89a4a9c0a4a604f415474/66b2ecc6497c13806dedf5f57a7303442159ace464b89a4a9c0a4a604f415474-json.log
[2021-07-31 16:56:27,654] Waiting on container to exit.
[2021-07-31 16:56:27,665] Creating exec into container.
[2021-07-31 16:56:27,676] Starting container.
[2021-07-31 16:56:27,687] Starting exec into container and waiting for exec to exit.
[2021-07-31 16:56:27,825] Adding container log: /var/lib/docker/containers/04e2d274a04ef549ebc031f3d7d56f0c93fcff9917ee7484ce18195e1092d1f8/04e2d274a04ef549ebc031f3d7d56f0c93fcff9917ee7484ce18195e1092d1f8-json.log
[2021-07-31 16:56:27,827] Waiting on container to exit.
[2021-07-31 16:56:27,941] Appending log line to log: {e77a0b33-4136-418e-89ca-7334e386bd2a}.
[2021-07-31 16:56:27,943] Appending log line to main log.
[2021-07-31 16:56:28,084] Container has state (exitCode: Some(1), OOMKilled Some(false))
[2021-07-31 16:56:28,100] Removing container build

[lukas]

I have the same error. Is there some solution?