apt-get in docker build fails on bitbucket pipeline (used to work)

Question

We have a couple of projects that uses our own docker images for testing packaging etc. The Dockerfile is checked in in the same repo and the first task of the pipeline is to build that image so that we can run the rest of the stuff inside containers of that image. This has been working for a while, but last week we noticed that the image didn't build anymore. It's failing on `apt-get update` I put together a small example: LOG: + docker build --progress=plain -t lambdas.test:latest -f ./test.Dockerfile .Sending build context to Docker daemon 212.5kBStep 1/2 : FROM ubuntu:22.0422.04: Pulling from library/ubuntub9e826c39a51: Pulling fs layerb9e826c39a51: Download completeb9e826c39a51: Pull completeDigest: sha256:9fd089c601e4ce6b61bfbba987b63ec6b73eb4ef9f568ad38b58cf0bbb019bffStatus: Downloaded newer image for ubuntu:22.04---> 41ba606c8ab9Step 2/2 : RUN apt-get update---> Running in 6fd16c9342c4Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [90.7 kB]Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [90.7 kB]Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [90.7 kB]Get:5 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [135 kB]Get:6 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [269 kB]Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1806 kB]Fetched 20.2 MB in 3s (7601 kB/s)Reading package lists...E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'E: Sub-process returned an error codeThe command '/bin/sh -c apt-get update' returned a non-zero code: 1002022-03-07T06:03:10.88920723Z stdout P ./test.Dockerfile: FROM ubuntu:22.04 RUN apt-get update Pipeline file: image: ubuntu:22.04 pipelines: default: - step: name: "build" caches: - docker services: - docker script: - docker build --progress=plain -t lambdas.test:latest -f ./test.Dockerfile .

Patrik S · Answer

Hello all,

The docker daemon version of Bitbucket pipelines was already updated to version 20.10.15 which should fix the incompatibility issue with apt-get update in Ubuntu 22 images.

In case you are still facing issues, please let us know or feel free to raise a new question here in community.

Thank you!

Kind regards,

Patrik S

Gavin Smith_eieptd · Answer

This issue looks to have been resolved when I tried it (again) on 9th June, 2022.

Taylor Phebillo · Answer

I'm pretty sure the underlying problem is that Ubuntu 22 uses an updated glibc version. The updated glibc version supports the new "clone3" system call. apt uses the clone3 system call if it's available (otherwise it falls back to a different clone version I think). Docker on bitbucket sees the system call, and because it's an old version of docker (before 20.10.10), it has no idea what the system call is. The default security policy is to block the system call, and (I think) returns EPERM. Now apt-get sees EPERM. It knows how to handle ENOSYS- it would fall back in that case. But on EPERM, it just fails. It doesn't seem to explain well why it fails- it seems to think the problem is with the command that was executed, rather than it's ability to start the command.

For potential solutions:

* Update docker on bitbucket: My vote
* Change security policy on bitbucket's docker image: Bitbucket sees what we're doing, and explicitly prohibits this. We got a fancy error message and everything.
* Change glibc: sounds hard to avoid breaking something else to me
* use LD_PRELOAD to have the clone3 system call just return ENOSYS: I actually think this might work, but my friend who is the one actually using this project doesn't seem interested. To be fair, I also think LD_PRELOAD is almost always a bad idea, I just think it's a fun trick.
* Just give up on bitbucket and build locally: Seems like the easiest option, which is what he's going with.

See launchpad bug 1943049