You're on your way to the next level! Join the Kudos program to earn points and save your progress.
Level 1: Seed
25 / 150 points
1 badge earned
Challenges come and go, but your rewards stay with you. Do more to earn more!
What goes around comes around! Share the love by gifting kudos to your peers.
Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!
Join now to unlock these features and more
I'm trying to create a pipe for customers that builds software for a niche platform, but I'm running into an issue.
One of the inputs for the pipe is the version of the software they're using. That translates to a Docker image tag that we have hosted on GitHub Container Registry (GHCR.io). So, at runtime, I need the pipe to read the version that the user has requested in the inputs, pull down a Docker image specific to that version, and then run it.
The problem I'm running into is that it seems that BitBucket doesn't allow Docker-in-Docker, so my pipe is unable to pull down or run the image. I've tried mounting /var/run/docker.sock when running my unit tests, I've tried running it as privileged, I've tried the DOCKER_BUILDKIT environment variable (both 1 and 0) and I get errors such as:
"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
"Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed."
"dial unix /var/run/docker.sock: connect: permission denied"
Has anyone built a custom pipe for BitBucket Pipelines that dynamically pulls down Docker images based on inputs to the pipe? What am I missing here?
@Paul Bromwell Jr_ hi. Maybe some additional info. The variable BITBUCKET_DOCKER_HOST_INTERNAL is available by default (also , i think DOCKER_HOST too). Below is a shell example how pipe docker container started to run with add-host parameters:
internal host example in pipeline -> --add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL",
docker-host-env example in pipeline -> --env=DOCKER_HOST="tcp://host.docker.internal:2375"
If you are using python then try this inside your pipe:
# first install docker via pip
docker_client = docker.from_env()
"DOCKER_HOST": "tcp://host.docker.internal:2375" # it should be available by default, but you could explicitly set this here
If you use bash then try this inside your pipe:
docker run ... --env=DOCKER_HOST="tcp://host.docker.internal:2375" \ --add-host="host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL" \ ...
Hello @Paul Bromwell Jr_ and welcome to the Community!
The docker daemon should indeed be accessible from the pipe's container, however the pipelines docker daemon does not use the UNIX socket /var/run/docker.sock to connect with the daemon and instead uses the TCP protocol on port 2375. You can confirm that by checking the start command logs of any pipe (pipes are essentially a docker container) :
pipe: <pipe image> + docker container run \ [...] --env=DOCKER_HOST="tcp://host.docker.internal:2375" \ [...]
This means the docker daemon should be available in the path tcp://host.docker.internal:2375.
With that in mind, if you would like to execute docker commands in your pipe, you need to either :
docker -H tcp://host.docker.internal:2375 pull ubuntu:latest
Hope that helps! Let me know in case you have any questions.
Thank you, @Paul Bromwell Jr_ !
Thanks so much for the info! Unfortunately, it doesn't seem like that's working. I added this line to the top of my Dockerfile that's being used to generate the pipe:
And I made sure all of my subprocess.Popen calls contained:
So all environment variables provided to the Python code should be passed to Docker CLI, and I'm getting this error:
error during connect: Post "http://host.docker.internal:2375/v1.24/images/create?fromImage=<REDACTED>&tag=<REDACTED>": dial tcp: lookup host.docker.internal on 10.5.0.2:53: no such host
Any ideas on that one?
Hello @Paul Bromwell Jr_ ,
It seems like additionally to
you should also provide:
as in the following example of one of the Atlassian-developed pipes:
and referenced on docker-py documentation :
No dice, I'm afraid. I still get this error even when adding to the /etc/hosts file (which is what extra_hosts is doing):
dial tcp: lookup host.docker.internal on 10.5.0.2:53: no such host
I did notice that if I have my Docker image run a shell script and run printenv (omitting the values I set), this is what it comes up with:
So I don't think that BITBUCKET_DOCKER_HOST_INTERNAL environment variable is getting passed to the containers.
I'm just going to rework the code to not have to use Docker-in-Docker.