Trouble running nextflow (nf-core) pipeline - docker authorization denied

[Michael Beavitt]

I'm working on a CI nextflow pipeline for some bioinformatics projects. The way it works, is it spawns a docker container for each step of the pipeline to complete the job, and then moves onto the next one. Everything happens in the ./work directory.

For some reason, it is being denied the ability to create new containers in Bitbucket cloud's workspace:

docker: Error response from daemon: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories.

But it seems like I satisfy those requirements:

echo $PWD

/opt/atlassian/pipelines/agent/build

echo $BITBUCKET_CLONE_DIR

/opt/atlassian/pipelines/agent/build

Can someone help me figure out what's going on?

[Theodora Boudale · Accepted solution]

Hi Michael and welcome to the community!

The argument -v in the docker run command is used to mount directories into the container that starts with this command:

• https://docs.docker.com/engine/reference/commandline/run/#volume

For Pipelines builds that run on Atlassian's infrastructure, we don't allow mapping volumes with a source outside $BITBUCKET_CLONE_DIR. You can find a list of all restricted commands here:

• https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/#Running-Docker-commands

You will need to check the docker run command that starts these containers or the docker-compose.yml in case you use Compose and only mount directories with a source inside $BITBUCKET_CLONE_DIR.

We have a feature request for allowing mounting volumes outside $BITBUCKET_CLONE_DIR: https://jira.atlassian.com/browse/BCLOUD-20986

This restriction does not apply to the self-hosted pipeline runners. If you have a Linux machine where you can run your builds, you could also use a Linux Docker Runner:

• https://support.atlassian.com/bitbucket-cloud/docs/runners/

Kind regards,
Theodora

[Michael Beavitt]

Thanks Theodora,

Unfortunately it's not possible to use any of the company machines as a runner, but I figured out the issue. 

Nextflow was trying to use 'user emulation' in docker, which meant mounting /etc/passwd and some other root directories, and I was able to turn off this feature in the config file.

It runs fine now!

They've also updated nf-core after an issue was raised to remove this default behaviour so this shouldn't be an issue in future..!

[Theodora Boudale]

Thank you for the update Michael, it's good to hear that the issue is resolved now!

Please feel free to reach out if you ever need anything else!