Pipeline broken after update to v25.0.2-multiarch-prod-stable

[Sam Redway]

Docker pipeline now not able to build. It breaks at the step where docker service is defined. ie my pipeline looks like this:

steps:
  - step: &run_unit_tests
      runs-on:
        - self.hosted
        - linux
      name: Run Unit Tests
      oidc: true
      size: 8x
      services:
        - docker
      script:
        - ...

And I dont reach the script it breaks on build step. If I remove teh lines which do services: docker then it does run through (but obviously I dont have access to docker deamon in pipeline).

This broke for no reason this morning. 

I have this in the docker terminal:

Runner warnings:

3

docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v25.0.2-multiarch-prod-stable: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.

4

5

/usr/local/bin/runit.sh: line 7: pipelines: not found

6

ip: can't find device 'nf_tables'

7

modprobe: can't change directory to '/lib/modules': No such file or directory

8

ip: can't find device 'ip_tables'

9

modprobe: can't change directory to '/lib/modules': No such file or directory

10

iptables: Failed to initialize nft: Protocol not supported

11

I checked and there is a new image when I do docker images called:

docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:  v25.0.2-multiarch-prod-stable

The old one had tag:  v20.10.24-multiarch-prod-stable

I'm guessing this update (created date is 8 hours ago the same time roughly as my pipeline broke) is what caused the pipeline to break. Although this could be a red herring!  Does anyone else have this issue? Is there a fix/ workaround?

[Theodora Boudale · Accepted solution]

Hi Sam and welcome to the community!

This issue seems to be related to the update of the Docker service. Our engineering team is looking into it.

In the meantime, since you are using a self-hosted runner, you can specify the image used by the docker service by adding the following in your yml file:

definitions: 
  services: 
    docker:
      image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.24-multiarch-runc-patch-prod-stable

If you are still experiencing issues, you can reach out via the support ticket that I see you created.

Kind regards,
Theodora

[cmorris98]

Thank you, this fixed our issue and our build pipelines are back up an running.

[Theodora Boudale]

That's good to hear @cmorris98, thank you for the update!

[Sam Redway]

This helped me get it back up and running and actually found I could now get it work with latest image by creating this in my top level defintions 

definitions:
  services:
    docker-self-hosted:
      image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v25.0.2-multiarch-prod-stable
      type: docker
    docker-not-self-hosted:
      memory: 7128
      type: docker

[Piotr Stuglik]

How exactly is this solved? A workaround is not a solution.

[Theodora Boudale]

Hi everyone,

Our engineering team has published a community article with some of the limitations that the upgrade of the Docker Engine introduced:

• https://community.atlassian.com/t5/Bitbucket-articles/Recent-docker-service-upgrade-in-Bitbucket-Pipelines/ba-p/2603987

You can read this article about the different limitations and what action needs to be taken in each case.

If you are still experiencing issues after following the suggestions in the article:

• for workspaces on Standard or Premium plan you can create a support ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.
  
  
• for workspaces on the Free plan, please create a new question in community including relevant errors you see and we will look into it. Please create a new question instead of posting a reply here, as this question will become convoluted and difficult to follow if we try to troubleshoot multiple users' potentially different issues.

Kind regards,
Theodora

[DP Developer]

Hi Team,

Help me for this issue

rpc error: code = Unknown desc = failed to pull and unpack image "docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable": failed to resolve reference "docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable": failed to do request: Head "https://docker-public.packages.atlassian.com/v2/sox/atlassian/bitbucket-pipelines-auth-proxy/manifests/prod-stable": read tcp 10.5.65.84:54922->204.246.191.16:443: read: connection reset by peer

[Tiago Jesus]

Runner warnings:

    docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.24-multiarch-runc-patch-prod-stable: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.

Cannot fix it with the older version. How solve this?

[Piotr Stuglik]

Is it still broken for you despite designating a specific dind image by any chance?

[Tiago Jesus]

Still broken with older version too, on my case.

[Piotr Stuglik]

New image produces the following log:

The old image (when set in the pipeline config file) does not produce said error.

But both produces the following:

Is that your case, too?

[Piotr Stuglik]

Two runs with the Docker image tag pinned produces different SHA values in the logs:

[Tiago Jesus]

I have two machines:

- If i run with on Docker version 20.10.25, build b82b9f3 no problem.

- I have problems with Docker version 24+

[Tiago Jesus]

Docker version 20.10.25, build b82b9f3 can run self hosted runners.

Docker version 24+ i have problems, cannot run.

[Piotr Stuglik]

I've tried pinning the image tag to the previous one, and while the following error disappeared from the Docker logs, the steps leveraging Docker still fails on `docker: command not found`:

`cat: can't open '/proc/net/arp_tables_names': No such file or directory`

I have discovered in the logs that the tag has two different SHA values between the failing and passing builds.

[slowquery]

I suddenly have a problem with the pipeline. I downgraded the image and temporarily solved it now, but I would appreciate it if you could reply as soon as the problem is resolved.

[Theodora Boudale]

Hi @slowquery, it's good to hear that the workaround solved the issue. I will post here when the issues are fixed.

[cmorris98]

Yes exact same issue!!  Would love to know how to force it to use the old version. If anyone figures this out please please post here. Our entire build process for both test and production pipelines are totally down.