apt-get in docker build fails on bitbucket pipeline (used to work)

[Pelle Ullberg]

We have a couple of projects that uses our own docker images for testing packaging etc. The Dockerfile is checked in in the same repo and the first task of the pipeline is to build that image so that we can run the rest of the stuff inside containers of that image. This has been working for a while, but last week we noticed that the image didn't build anymore. It's failing on `apt-get update`

I put together a small example: 

LOG:

+ docker build --progress=plain -t lambdas.test:latest -f ./test.Dockerfile .

Sending build context to Docker daemon 212.5kB

Step 1/2 : FROM ubuntu:22.04

22.04: Pulling from library/ubuntu

b9e826c39a51: Pulling fs layer

b9e826c39a51: Download complete

b9e826c39a51: Pull complete

Digest: sha256:9fd089c601e4ce6b61bfbba987b63ec6b73eb4ef9f568ad38b58cf0bbb019bff

Status: Downloaded newer image for ubuntu:22.04

---> 41ba606c8ab9

Step 2/2 : RUN apt-get update

---> Running in 6fd16c9342c4

Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [90.7 kB]

Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]

Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [90.7 kB]

Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [90.7 kB]

Get:5 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [135 kB]

Get:6 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [269 kB]

Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]

Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1806 kB]

Fetched 20.2 MB in 3s (7601 kB/s)

Reading package lists...

E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'

E: Sub-process returned an error code

The command '/bin/sh -c apt-get update' returned a non-zero code: 100

2022-03-07T06:03:10.88920723Z stdout P

./test.Dockerfile:

FROM ubuntu:22.04
RUN apt-get update

Pipeline file:

image: ubuntu:22.04
pipelines:
  default:
    - step:
        name: "build"
        caches:
          - docker
        services:
          - docker
        script:
          - docker build --progress=plain -t lambdas.test:latest -f ./test.Dockerfile .

[Taylor Phebillo]

I'm pretty sure the underlying problem is that Ubuntu 22 uses an updated glibc version. The updated glibc version supports the new "clone3" system call. apt uses the clone3 system call if it's available (otherwise it falls back to a different clone version I think). Docker on bitbucket sees the system call, and because it's an old version of docker (before 20.10.10), it has no idea what the system call is. The default security policy is to block the system call, and (I think) returns EPERM. Now apt-get sees EPERM. It knows how to handle ENOSYS- it would fall back in that case. But on EPERM, it just fails. It doesn't seem to explain well why it fails- it seems to think the problem is with the command that was executed, rather than it's ability to start the command.

For potential solutions:

• Update docker on bitbucket: My vote
• Change security policy on bitbucket's docker image: Bitbucket sees what we're doing, and explicitly prohibits this. We got a fancy error message and everything.
• Change glibc: sounds hard to avoid breaking something else to me
• use LD_PRELOAD to have the clone3 system call just return ENOSYS: I actually think this might work, but my friend who is the one actually using this project doesn't seem interested. To be fair, I also think LD_PRELOAD is almost always a bad idea, I just think it's a fun trick.
• Just give up on bitbucket and build locally: Seems like the easiest option, which is what he's going with.

See launchpad bug 1943049

[alex udrescu]

I'm seeing the same behavior in pipelines for our `ubuntu:22.04` based images.  `ubuntu:20.04` and `ubuntu:21:10` still work however (for now).

[Patrik S]

Hello all,

The docker daemon version of Bitbucket pipelines was already updated to version 20.10.15 which should fix the incompatibility issue with apt-get update in Ubuntu 22 images.

In case you are still facing issues, please let us know or feel free to raise a new question here in community.

Thank you!

Kind regards,

Patrik S

[Gavin Smith]

This issue looks to have been resolved when I tried it (again) on 9th June, 2022.

[cyrilcanonico]

Hello !

I faced the same issue today (in other circumstances but this might help as I had the exact same result)

Can you check the docker version you are running this on?

I had the issue while running docker-ce version 20.10.8, I upgraded to the lateset one (version 20.10.12 as of today), and everything went well after this.

Hope this helps.

Have a nice day !