Issue with buildkit started today

[Ricardo José Brüggemann]

Our pipelines were working fine till ~9:40 today. 

After that, self hosted runners and bb runners are with issues to build:

BB Runner:

+ docker build -t "$IMAGE" .

2

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.

3

BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0

4

environment-variable.

5

6

Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?

------------------------------------------------------------------------

Self hosted runner: 

Login Succeeded

133

#0 building with "default" instance using docker-container driver

134

135

#1 [internal] booting buildkit

136

#1 pulling image moby/buildkit:buildx-stable-1

137

#1 pulling image moby/buildkit:buildx-stable-1 4.7s done

138

#1 creating container buildx_buildkit_default 0.0s done

139

#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

140

------

141

> [internal] booting buildkit:

142

------

143

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

144

ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

145

35 v0.11.2 /usr/local/libexec/docker/cli-plugins/docker-buildx buildx build --target test --build-arg BITBUCKET_USERNAME=osprey-renovate-bot --build-arg APP_PASSWORD=$PIPELINE_APP_PASSWORD -f docker/app/Dockerfile . -t ums:rm-834

146

github.com/docker/docker/client.(*Client).checkResponseErr

147

github.com/docker/docker@v24.0.5-0.20230714235725-36e9e796c6fc+incompatible/client/request.go:240

148

github.com/docker/docker/client.(*Client).sendRequest

149

github.com/docker/docker@v24.0.5-0.20230714235725-36e9e796c6fc+incompatible/client/request.go:126

150

github.com/docker/docker/client.(*Client).post

151

github.com/docker/docker@v24.0.5-0.20230714235725-36e9e796c6fc+incompatible/client/request.go:45

152

github.com/docker/docker/client.(*Client).ContainerCreate

153

github.com/docker/docker@v24.0.5-0.20230714235725-36e9e796c6fc+incompatible/client/container_create.go:63

154

github.com/docker/buildx/driver/docker-container.(*Driver).create.func3

155

github.com/docker/buildx/driver/docker-container/driver.go:150

156

github.com/docker/buildx/util/progress.(*subLogger).Wrap

157

github.com/docker/buildx/util/progress/progress.go:79

158

github.com/docker/buildx/driver/docker-container.(*Driver).create

159

github.com/docker/buildx/driver/docker-container/driver.go:113

160

github.com/docker/buildx/driver/docker-container.(*Driver).Bootstrap.func1

161

github.com/docker/buildx/driver/docker-container/driver.go:59

162

github.com/docker/buildx/util/progress.Wrap

163

github.com/docker/buildx/util/progress/progress.go:47

164

github.com/docker/buildx/driver/docker-container.(*Driver).Bootstrap

165

github.com/docker/buildx/driver/docker-container/driver.go:55

166

github.com/docker/buildx/driver.Boot

167

github.com/docker/buildx/driver/driver.go:79

168

github.com/docker/buildx/build.ensureBooted.func1.1

169

github.com/docker/buildx/build/build.go:177

170

golang.org/x/sync/errgroup.(*Group).Go.func1

171

golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75

172

runtime.goexit

173

runtime/asm_amd64.s:1598

Any idea why this started ?

[Erwan d_Orgeville]

Hi everyone,

After exchanging with @Farha Atbai through Bitbucket's cloud support (thanks @Theodora Boudale for the link), here is the conclusion we came to:

1. If you are using an old version of docker-compose to interface with the docker daemon, you'll have to upgrade. I had to upgrade from `1.8.0-rc2` to `v2.24.5`

2. With this newer docker engine that Bitbucket rolled out, you have to disable the new "Docker Buildkit" system. Do so by adding the following to your pipelines, before executing anything docker related: `export DOCKER_BUILDKIT=0`

I hope this helps other users.

Tagging everyone else on this thread for visibility: @Ricardo José Brüggemann @Vegard Theriault @Chantal Delachevrotiere 

[Chantal Delachevrotiere]

Similar issue here
`cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 127.0.0.1:2375: connect: connection refused`

[Theodora Boudale]

Hi Ricardo José and welcome to the community!

These errors may be related to an update we started rolling out, specifically an update to the Docker service and an upgraded version of the Docker client for running docker commands from 20.10.24 to 25.0.2:

• https://confluence.atlassian.com/bbkb/infrastructure-changes-in-bitbucket-pipelines-1142431176.html

I would suggest creating a ticket with the support team for further investigation and options regarding resolution.

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.

Kind regards,
Theodora

[Ricardo José Brüggemann]

Thanks Theodora, will open a ticket there. 

[Vegard Theriault]

We’re having the same issue

[Erwan d_Orgeville]

Similar issue here. Our "docker-in-docker" builds broke overnight. Now, we're greeted with a "Couldn't connect to Docker daemon at http://localhost:2375 - is it running?" error every time we try calling `docker-compose build` :(