Using docker compose build ssh with BuildKit when --mount=type=ssh is disabled

[Seth Miller]

Since `RUN --mount=type=ssh` is disabled, how can I configure SSH with the running ssh-agent in my CI container? We are using the common docker-in-docker setup of running our own CI container and having docker compose build the image for us. It is failing due to your unfortunate limitations:

```

service:
    build:
        context: .
        ssh:
          - default

```

and the Dockerfile requires a
```

RUN --mount=type=ssh pip install [...]
```

Produces:

```

docker compose run app ./util/lint

+ docker compose run app ./util/lint

Network build_default Creating

Network build_default Created

app Pulling

app Warning

#1 [internal] booting buildkit

#1 pulling image moby/buildkit:buildx-stable-1

#1 pulling image moby/buildkit:buildx-stable-1 2.7s done

#1 creating container buildx_buildkit_default done

#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

------

> [internal] booting buildkit:

------

Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
```

Which is a bit silly. How are we supposed to work around these limitations?

Also how far out is the plan to ditch Kubernetes so we can have features your competitors offer like not having to work around trivial things and mutli-arch builds? That would be great!

No Markdown formatting for a product who's only made for developers? Another oops.

[Seth Miller · Accepted solution]

Bitbucket Pipelines does not support modern BuildX environments and therefore many features of Docker Compose V2 are unavailable. Since Docker Compose V1 is EOL in June 2023,

Our choices are:

1. Switch to a more modern competitor like GitLab or Github
2. Wait for https://jira.atlassian.com/browse/BCLOUD-22599 to be implemented at Atlassians leisure.

[Patrik S]

Hello @Seth Miller ,

Thank you for reaching out to Atlassian Community!

While using Bitbucket Pipelines with Docker BuildKit enabled, that are some restrictions applied due to security reasons, which are listed in Docker BuildKit restrictions.

One of those restrictions is that the directive RUN --mount=type=ssh  is currently disabled in the pipeline environment. However, you can use the --ssh argument to provide the SSH key to the docker command, as in the below example : 

--ssh default=$BITBUCKET_SSH_KEY_FILE

The BITBUCKET_SSH_KEY_FILE is a default variable that contains the path of the SSH key configured in your pipelines (Repository Settings > SSH keys under pipelines).  This should allow you to use that key when connecting to external services.

As for allowing building multi-architecture images in pipelines, we do have a feature request open to implementing that functionality : 

• https://jira.atlassian.com/browse/BCLOUD-15317

We encourage you to add your Vote and mark yourself as a Watcher in that feature ticket so you will be notified whenever there's any update.

As a workaround, while that feature is not available, it's possible to build multi-architectural images when using Linux Docker self-hosted runners, as they allow you to configure a custom docker-in-docker service. Our engineering team has posted an example of using runners to build multi-arch images in this comment of that feature request.

Thank you, @Seth Miller !

Patrik S

[Seth Miller]

Patrik,

Thanks for your response and clarification. It's a shame we can't use many of those best practices features but it is what it is.

I might actually have sent us down a red herring path. I can't get BuildKit to run at all. I think it's due to auto buildx inclusion??? Do you know how I can use BuildKit properly? I see the announcement but I'm unable to get it to work. I believe I've stumbled upon this related issue: https://community.atlassian.com/t5/Jira-Work-Management-Questions/Can-t-run-compose-in-bitbucket-pipelines-getting-privileged-true/qaq-p/2233411

Perhaps this as well: https://jira.atlassian.com/browse/BCLOUD-22599?filter=98686

I'm not trying anything fancy, just docker compose build.

Thanks!

[Seth Miller]

@Patrik S- I'm still looking for an update on this. Docker Compose V1 seems to be the only supported version of Docker Compose on Pipelines and it's EOL in June of 2023. What does your roadmap look like concerning this?

Thanks

[Patrik S]

Hello @Seth Miller ,

When enabling BuildKit and using Docker Compose V2, the compose build command will use by default buildx, which is currently not supported in pipelines (see Docker BuildKit restrictions and BCLOUD-22066 ).

Currently, we don't have a provision to disable buildx only while keeping the BuildKit enabled. To solve this issue, you may need to downgrade the docker-compose version to 1.29.2. 

If you want to keep using Docker Compose V2, you would have to disable BuildKit, so compose build will not try to use buildx. The following command can be used to disable BuildKit in the pipelines environment : 

export DOCKER_BUILDKIT=0

We also have an open feature request to implement an option to only disable BuildX for the Docker Compose Version 2, while keeping the BuildKit enabled. You can access the feature using the link below : 

• https://jira.atlassian.com/browse/BCLOUD-22599

I see you have already voted for that feature request and added yourself as a watcher, so whenever there's any update on that feature you will be notified. Please note that all features are implemented with this policy in mind.

Thank you, @Seth Miller !

Patrik S

[Seth Miller]

@Patrik S,

Thank you for your response. It seems like the Docker team might be partially at fault here too as they seem to be abandoning non-buildx environments. I appreciate you chiming in. I'll rally as much support as I can for that feature!

Have a great day!

-Seth

[Huacong Cai]

Hi @Patrik S, could you elaborate on how to use the SSH key just with --ssh argument and without RUN --mount=type=ssh?

From all the examples I found so far, I need to use both. Set the ssh key file location through  --ssh while running docker build, and let the docker command inside Dockerfile to use it through RUN --mount=type=ssh

[Patrik S]

Hello @Huacong Cai ,

You should currently be able to use the RUN --mount=type=ssh directive in pipelines along with docker build command as in the following example :

Dockerfile example : 

FROM atlassian/default-image:4

RUN  --mount=type=ssh ssh -o StrictHostKeyChecking=no -Tvvv git@bitbucket.org

Docker build command : 

DOCKER_BUILDKIT=1 docker build --ssh default=$BITBUCKET_SSH_KEY_FILE -t myaccount/myimage:tag .

I have also raised an internal request to have the documentation updated, as it currently says the usage of SSH mount is still restricted, which is no longer the case.

Thank you, @Huacong Cai !

Patrik S

[christian_campanaemard]

Thanks @Patrik S for the update! Just a friendly reminder that the documentation is still not updated here: Run Docker commands in Bitbucket Pipelines | Bitbucket Cloud | Atlassian Support

I'm glad I found this comment before attempting to find a workaround to this formerly disabled feature. This feature is essential, and the outdated documentation may mislead customers to look at alternative solutions that would require significantly more effort to implement.

[Seth Miller]

@Patrik S- If this is resolved now care to summarize it in a new answer and I can accept that?

[Riley Venable]

To configure SSH with the running ssh-agent in your CI container when --mount=type=ssh is disabled in Docker Compose, you can copy your SSH keys into the container during the build process. You can use the Dockerfile example provided in the context, which copies the `id_rsa` and `id_rsa.pub` files into the container's `/root/.ssh` directory, and configures SSH to use the `id_rsa` key. It also installs `openssh-client` and starts the `ssh-agent` in the container. When you run your container, you can then add your SSH key to the agent using the `ssh-add` command:

```
ssh-add /root/.ssh/id_rsa
```

As for your second question, Atlassian has not announced any plans to ditch Kubernetes or offer multi-arch builds that competitors already have.

[Seth Miller]

> you can copy your SSH keys into the container during the build process.

That's my trouble is that the keys are baked into the image forever and I was doing all sorts of funky multi-stage build stuff to ensure that didn't happen. My Dockerfile was looking pretty complex.