Occasionally Pipeline is unable to connect to AuthZPlugin

[Christian Geisler]

Occasionally a pipeline fails due to a failure to connect to the pluginAuthZPlugin.AuthZRes. Here is the docker error log:

time="2019-11-24T14:26:50.782904282Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZRes: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 1s"time="2019-11-24T14:26:51.783581826Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZRes: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 2s"time="2019-11-24T14:26:53.788686520Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZRes: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 4s"time="2019-11-24T14:26:57.789059713Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZRes: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 8s"time="2019-11-24T14:27:05.789590970Z" level=error msg="AuthZResponse for POST /v1.25/build?t=hdp_amphora_local_django&q=False&nocache=False&rm=True&forcerm=False&pull=False&dockerfile=.%2Fcompose%2Flocal%2Fdjango%2FDockerfile returned error: plugin pipelines failed with error: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused"time="2019-11-24T14:27:05.789656585Z" level=error msg="Handler for POST /v1.25/build returned error: plugin pipelines failed with error: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZRes: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused"2019-11-24 14:27:05.789696 I | http: multiple response.WriteHeader callstime="2019-11-24T14:27:05.796135397Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZReq: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 1s"time="2019-11-24T14:27:06.797576578Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZReq: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 2s"time="2019-11-24T14:27:08.797997268Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZReq: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 4s"time="2019-11-24T14:27:12.798347438Z" level=warning msg="Unable to connect to plugin: /run/docker/plugins/pipelines.sock/AuthZPlugin.AuthZReq: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused, retrying in 8s"time="2019-11-24T14:27:20.798690871Z" level=error msg="AuthZRequest for GET /v1.25/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dbuild%22%2C+%22com.docker.compose.service%3Ddjango%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D returned error: plugin pipelines failed with error: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused"time="2019-11-24T14:27:20.798740006Z" level=error msg="Handler for GET /v1.25/containers/json returned error: plugin pipelines failed with error: Post http://%2Frun%2Fdocker%2Fplugins%2Fpipelines.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/pipelines.sock: connect: connection refused"

Googling did not help much. Anybody any ideas?

[Matthias Prieth]

Are there any updates on this issue? It happens very occasionally with `docker tag` or `docker push` commands. Can I provide some more logs or something else to help?

I don't know what to do here, as it happens very randomly & sometimes we can't deploy for several hours. So for us it's a serious blocker at the moment.

[Igor Filipek]

we are having the same issue, fails randomly.

[Tommi Laukkanen]

Same error with Bitbucket Pipelines Pipe: AWS ECR push image.

[Matthias Prieth]

I don't know if it's a fix or if Atlassian fixed something, but since I added the following to my bitbucket-pipelines.yml file I did not get this error anymore:

definitions:
  services:
    docker:      
      memory: 2048

[Sascha Kohnke]

this fixed it for me as well

[Batbayar_B]

I have following values and got this error sometimes.

definitions:
  services:
    docker:
      memory: 3072

[etiennecaldichourypys]

same here. Would be great to have a fix, we're loosing plenty of times at each fail.

[Andres Davila]

C'mon guys pls say something! We suppose to be a community, not just a wailing wall... 

[Javier Vargas]

We are facing the same problem

[Diler96]

Same. Can't push image

[Max Mönch]

Seems to be still a problem....

Our tests are not that big that is should be a problem with 2048 memory. 

This is really a waist of time, have to build 5-10 times until it get trough.

[Asma Elfaleh]

I have the same problem here

[Maycon Dos Santos Nunes]

Almost one year without any answer... I also have the same problem here

[Deleted user]

Same problem here

[Deleted user]

happens intermittently but it's more or less on memory/cache issue (most of the times), try to re-run or increase the memory

[Rick Zakharov]

Same error. Please help.

[Marc Salat]

Same error

[Eric Jansen]

Same error with sonarsource/sonarcloud-quality-gate:0.1.3

[Carlos Soza Canales]

Same error.

Please fix as soon as posible.

[Denis Vakulenko]

Same