Changes to make your containers more secure on Bitbucket Pipelines.

As a part of the ongoing security hardening process in Bitbucket Pipelines, we decided to enable docker userns remap to eliminate potential security risks that might affect our users.

This change affected a small percentage of people using docker in their builds. So, we wanted to share the 4 different limitations that this change introduced, and the actions required to make your builds work in the more secure system.

Issues caused by limitations of userns remap

1. Docker: network=host no longer supported

As mentioned in the docker documentation linked above, enabling docker userns remap introduces the following limitations:

The following standard Docker features are incompatible with running a Docker daemon with user namespaces enabled:

- sharing PID or NET namespaces with the host (--pid=host or --network=host).

- external (volume or storage) drivers which are unaware or incapable of using daemon user mappings.

- Using the --privileged mode flag on docker run without also specifying --userns=host.

In our previous security updates we disallowed pid=host and --privileged containers however network=host is a new limitation.

As network=host will no longer be allowed, there are a few other ways you can communicate between your code and containers.

To still allow your containers and code to communicate please see the following guidance.

If you need to communicate with a service running in docker from your build container, when starting the service provide it a port mapping using -p <host-port>:<container-port> you can then access the service using localhost:<host-port>.

If you need to communicate from a service running in docker to a service running in your build container, when starting the service provide it the following host entry using --add-host host.docker.internal:$BITBUCKET_DOCKER_HOST_INTERNAL you can then access the service using host.docker.internal:<port>.

If you need to talk between services running in docker, they have to be attached to the same docker network so you can address the services using <container-name>:<container-port> or consider using docker compose to run the services which creates a custom network for the compose stack.

2. Builds fails due to error: “Container ID Cannot Be Mapped to Host ID Error

Due to the way userns remap works (by remapping UID/GIDs to another less privileged UID/GID in the hosts namespace), UID/GIDs that are placed on files must be in the range 0-65535.

If you recieve this error, you will need to perform a fix on the image that has this invalid UID/GID.

If your recieved this error against a file that you created and placed into an image, the fix is to change the UID/GID of the file to one within the range.

To find the file, first get the invalid UID/GID from the error message and run the following command on the files within the container.

find / -uid|-gid <invalid-uid-or-guid> -ls

To fix the ownership of the file use the following command

chown -R <uid-in-range>:<gid-in-range> filename

If you received this error against a file that was in the base image you depend on, you will need to raise a support case and work with the image maintainers of the base image to get this resolved.

3. Test containers

Test containers recently released a fix to disable ryuk in version 1.10.6 by setting the environment variable TESTCONTAINERS_RYUK_DISABLED to true as documented here. In order to keep using test containers in pipelines you will need to upgrade your test containers dependency to this version or greater and set the environment variable in your pipelines (either directly in the step, or via an account/repository variable).

4. Docker build: removing files doesn’t remove them

Due to bugs in docker, when files were modified (removed/alterered) during a docker build via RUN/COPY commands the changes weren't being reflected in subsequent layers.

We have upgraded docker to pull in the fix for these issues.

However some failures require user intervention, as some commands were modifying the UID/GID (similar to the error below). For these situations docker now supports --chown as an argument to some dockerfile commands.

https://github.com/moby/moby/issues/36648

https://github.com/moby/moby/issues/35709

https://github.com/moby/moby/issues/32919

https://github.com/moby/moby/issues/30110

1 comment

My build fails with error: Container ID 165586 cannot be mapped to a host ID

So, I guess I am falling into the second case. Although I tried your suggestion, I couldn't find any file with id `165586`. 

 

Any clue?

Comment

Log in or Sign up to comment
Community showcase
Published Nov 06, 2018 in Bitbucket

Upgrade Best Practices

Hello! My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible. Today, I want to bring the discussion that Jennifer, Matt, and ...

1,963 views 7 10
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you