Secure secrets in Docker build layers run within a Bitbucket Pipeline

Jonathan Little_cukkre

For one of our services, ideally we'd like to run the entire build process in a Docker container (in a multi-stage build) so that our build environment is controlled and shared between development computers and Pipelines. But, part of the build depends on credentials for access to a private package repository.

If we run the build directly in Pipelines, then we can use secured Pipelines environment variables and presume that the credentials are handled securely since that's presumably what that feature is designed for.

However if we run the build using Docker within pipelines (i.e. if the pipelines config includes a "docker build ..." command which runs the application build as part of a multi-stage Docker image build) then there will exist a cached Docker layer which contains the values of the any build args. Now, build args are not supposed to be used for secrets but disturbingly they still appear to be the easiest and most accessible way to pass in arguments to a build and I would guess a lot of people are actually using them this way. If cached images are getting stored only on customer-controlled build machines then this may be an acceptable risk.

What security can a user of Bitbucket Pipelines assume is applied to the actual Pipelines build environments and to caches (Docker or otherwise, but especially Docker) that are created as part of builds?

Philip Hodder

Hi Jonathan,

Thanks for being diligent with your security practices. And sorry for the delayed response.

I'll go into more details, but the short answer is that your concern won't be an issue, assuming you don't add any secrets as part of your docker build process.

Pulling the Docker Build Image

When pipelines runs your build inside the docker container, the variables are passed into the runtime. Similar to: 'docker run -e MY_VAR=MY_VAL image/name'. (Pipelines is running on Kubernetes, so this isn't exactly how it works, but is a close enough analogy). This doesn't generate any caches containing secrets. We cache image layers that are publicly available for performance benefits, but because we inject the variables in at runtime they are not cached with the image layers.

NOTE: These are not user configured caches. And thus, do not have a 'caches' section in the bitbucket-pipelines.yml.

You can verify the secrets are not in the system caches by removing them from your build, they will not be present in your build container (when running 'env') as we retrieve them directly from Pipelines storage.

Running Docker Build

Building a Docker container will interact with image layers in three ways.

1. By pulling existing layers remotely
2. By creating new layers
3. By referencing cached layers.

When building a new image, the newly built image layers are built on top of the image specified in your 'FROM' command. The pipelines build container is not included in this context. The only way you'll be to add your secrets to your build is if you explicitly add them to an image layer, such as with an 'ENV' command.

The docker cache (the one defined in the 'caches' section) inside of your build does not contain any information about your pipelines build image. It only includes image layers pulled/built from the Docker CLI running inside of your pipeline build.

If you'd like to verify this behaviour, you can run 'docker run --entrypoint env image/name' on an image you've built, to verify the content of the environment variables.

The above holds for multi-staged builds as well.

Feel free to reply with more questions/clarifications.

Thanks,

Phil