Recent docker service upgrade in Bitbucket Pipelines

[#1]

zOn 2 Feb 2024, as a response to a security upgrade released by Docker, we upgraded Docker Engine from 20.10.24 to 25.0.2 to ensure we maintain our high security standards.

This change affected a small percentage of users using docker in their builds. So, we wanted to share some of the different limitations that this change introduced, and the actions required to make your builds work in the new version. For more details, visit Docker Docs — Docker Engine release notes.

1. buildkit enabled as the default builder

Since docker engine 23, buildkit has become the default builder for docker build. Users who are using docker build will default to having buildkit enabled with the upgrade. Users can disable buildkit and using the legacy build by running export DOCKER_BUILDKIT=0 or setting a variable in pipelines.

2. Older docker client versions won’t work with docker engine 25

Due to incompatibilities with the newer docker API, some older docker client versions in the 19.x range have been found to have problems. User might see their build failed with following error:

failed to authorize: failed to fetch oauth token: Get "https://auth.registry.host/token?query_params": stopped after 10 redirects

We recommended using the docker client we provide in pipelines, or using docker client v20.10.24. At the moment, we don’t support docker clients greater than v20.10.24.

3. Older docker compose 1.x and 2.x versions won’t work with docker engine 25

Due to incompatibilities with the newer docker API, some older docker compose 1.x versions is not working with docker engine 25. User might see their build failed with following error:

Couldn't connect to Docker daemon at http://10.37.234.13:2375 - is it running?

Action: Users should upgrade docker compose version to the latest in the 1.x or 2.x range (at the time of writing is 1.29.2 or 2.24.5).

4. docker compose v2 won’t work with buildkit

Docker compose with buildkit builder by default require privileged mode, which was disabled by Bitbucket Pipelines for security reason. User might see following error with docker compose v2 cli:

ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

Action: Users should disable buildkit with export DOCKER_BUILDKIT=0 .

5. ca-certificates required in build image by buildkit

buildkit require to do additional checks with registry, for users using docker build with images that don’t have ca-certificates installed (i.e. ubuntu:latest), their build would fail with the following error:

failed to authorize: failed to fetch anonymous token: Get "https://register.host/token/?query_params": x509: certificate signed by unknown authority 

Action: Users should either install ca-certificates in their build script, or disable buildkit with export DOCKER_BUILDKIT=0 .

6. "ADD somefile.tar.gz destination" results in files created with owner nobody

When adding .tar.gz file to a docker image using the ADD Dockerfile directive, a bug in buildkit causes the owner of the files in the decompressed archive to be set to the user nobody. Bitbucket Pipelines uses a feature called userns-remapping to provide improved security without our cloud build environments, and buildkit incorrect assigns ownership to files in decompressed tar archives when this feature is enabled. 

This bug is tracked with buildkit at https://github.com/moby/buildkit/pull/4712 

Users may see the following error in the docker daemon logs:

Host ID 0 cannot be mapped to a container ID

To work about this problem, either:

Disable buildkit:

export DOCKER_BUILDKIT=0

Alternatively, decompress the archive outside of docker build and add the extracted files and folders using ADD or COPY:

mkdir unarchived && tar -xvzf my_archive.tar.gz -C ./unarchived

7. Some library like jib not supporting docker engine 25

For user using jib, they might see following error in their build:

Unrecognized field "LayerSources" (class com.google.cloud.tools.jib.docker.json.DockerManifestEntryTemplate)

User need to either wait for a new release of jib(related issue) or directly using docker build to build image or use self-hosted runners with a dind < 25.0.0, but be aware that DIND 24.0.9 doesn’t contain some of security fixes.

[Piotr Stuglik]

Does this resolve the `docker: command not found` issue?

I am currently trying the following:

"docker: Command not found" error while running docker commands in self-hosted Runner | Bitbucket Cloud | Atlassian Documentation

[Eduar Bastidas]

I guess it was this change that broke the pipelines of many users who use composer.
Check this issue link 

[Edmund Munday]

Hi @Eduar Bastidas - looking at that Github thread, it appears that Composer issue is being caused by changes made in Compose 2.7.0 and was resolved when the user pinned Composer to 2.6.6.

https://github.com/pestphp/pest/issues/94#issuecomment-1936233204

[Karthik]

We previously encountered and bitbucket team resolved an issue where the Docker command was not found in our pipelines earlier this month. However, as of today (February 14, 2024), we're experiencing the same issue again. Was there have been any recent changes, updates, or releases on your end that might have reinstated this problem? Any insights would be greatly appreciated.

[bishan.vithanage]

We have this issue, may be affected by this upgrade.

https://community.atlassian.com/t5/Bitbucket-questions/Unable-to-Run-Whitesource-Mend-Scan-on-Docker-Container-Images/qaq-p/2615233

[Piotr Stuglik]

The issue is back!

[Hrólfur Gylfason]

Is there any way to opt into this rollout? We would be interested in using the --cache-from and --cache-to to use AWS ECR as a cache but those don't seem to be available in docker 20, which our pipeline docker service is still on.

[Chirag Jain]

It is May 2024 and yet this rollout has not completed, can we please have a choice to use this?

[moritz_matchory_com]

Is this *still* not completed?! I'm also looking for `--cache-to` and `--cache-from`, since your built-in Docker cache just... does not work for any BuildKit builds.

This is so bloody disappointing. Could you PLEASE consider updating your infrastructure faster than... YEARLY!?

[Denis Baltor]

We're facing an error in our pipeline whose message claims we're using Docker API v1.24, however I can read this in the pipeline docker's log:
```
time="2024-12-10T16:30:15.530144414Z" level=info msg="Docker daemon" commit=f417435 containerd-snapshotter=false storage-driver=overlay2 version=25.0.3
```
Any idea on what's going on?