Docker Buildkit error when use external caches

Alexander_pda0ju

Context:
We use Bitbucket Pipelines to build docker images. To optimise builds, we follow this guide - and specifically docker buildkit `--cache-from` approach:

docker build 
  --cache-from $TAG 
  --tag $TAG 
  --file Dockerfile 
  --build-arg BUILDKIT_INLINE_CACHE=1 .

Issue:
When we run a pipeline second time and external cache is presented in registry, we're getting pipeline System Error: "System service container 'docker' error: 'null'". It seems to be buildkit-related stack overflow `goroutine stack exceeds 1000000000-byte limit` issue mentioned in this comment - and this issue seems to be fixed in newer buildkit releases.

Output of build step:

#28 pulling sha256:1dcbb5745d79ac470b93ad64dc40bde6f8789faf4053163548e8182b9326f5d6 1.0s done
#28 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#28 pulling sha256:0e07f5708d2a064481f5a5ff358889468ab6c20c68a154b30aaf996d053bd4d1 0.3s done
#28 pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.3s done
#28 CACHED
unexpected EOF

Logs of docker service:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc021ea2388 stack=[0xc021ea2000, 0xc041ea2000]
fatal error: stack overflow
runtime stack:
runtime.throw({0x1b688c1?, 0x33c8e40?})
/usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x7f157affd0b8 sp=0x7f157affd088 pc=0x439bbf
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1103 +0x5cc fp=0x7f157affd270 sp=0x7f157affd0b8 pc=0x453f2c
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:570 +0x80 fp=0x7f157affd278 sp=0x7f157affd270 pc=0x46bf20
goroutine 274 [running]:
runtime.mapaccess2_fast64(0x210cd40, 0xc0010119e0, 0xc00068f6e0)
/usr/local/go/src/runtime/map_fast64.go:53 +0x18d fp=0xc021ea2398 sp=0xc021ea2390 pc=0x413d6d
github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc00068f6e0, 0xc041ea0df0)
/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1/utils.go:274 +0x4f fp=0xc021ea2508 sp=0xc021ea2398 pc=0x180c1ef
github.com/docker/docker/vendor/github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc00068f680, 0xc041ea0df0)

Questions:
1. Can this issue be fixed on BitBucket side by updating Docker or Buildkit version?

2. Do you have any suggestions / workarounds how we can fix it on our side and keep using external caches?

3. Is there a way to use newer version of docker in Pipelines executed in Atlassian Cloud? I tried to use `docker:dind` as build image, but it seems like it's not allowed to use docker daemon inside build container.

Patrik S

Hello @Alexander and welcome to the Community!

I'm afraid it's not possible to manually update the Docker version of pipelines that are run in Atlassian infrastructure, as this is controlled at the platform level.

However, you have the option to use Linux Self-hosted runners. With runners, you have more control over the environment and have the ability to use custom docker daemon, as described in the article Custom docker-in-docker image .

You might want to try testing with a local runner and a more recent version of docker daemon to check if the issue was fixed in the more recent version of docker.

I see that your workspace has an active subscription and you also opened an internal ticket for this issue. I would suggest focusing on that ticket to avoid any confusion that may arise from discussing the same issue in multiple places. Once the ticket is resolved, feel free to share the solution here on Community as it may help other users who might come across the same issue.

Thank you, @Alexander !

Patrik S