nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

rm -rf inside container via Bitbucket pipeline

Tania Petsouka

Hi,

we use bitbucket pipelines for our CD to Google cloud.

Among others, we build an apache image. Inside it's dockefile we have a step to remove default apache's vhosts:

RUN rm -rf /etc/apache2/sites-available/*

This worked fine until yesterday. Today the rm stopped working and after searching we found out that it's probably due to bitbucket's filesystem which is formatted without d_type support.

See here http://www.rubyrube.com/2017/06/28/dockerfile-run-rm-not-working-ex-run-rm-rf-usrlocaltomcatwebappsroot/

*We deployed the image outside the pipeline properly, so we're sure that it's related to bitbucket.

Can you advice a workaround for this?

Thanks

Accepted answers

Philip Hodder

Hello,

We've rolled out some changes that have fixed this issue. Can you please try run your docker builds again. Please comment if you're still having issues with deleting/editing files as part of your Docker build.

Thanks,

Phil

All comments

torauter_supro8

Hi Tania,

Do you have any news on that topic?

We experience the same issue. I already created a ticket.

As a workaround, you could use a customized base docker image, where you already deleted the vhosts.

Tobi

Tania Petsouka

Hi Tobias,

i sent a ticket as well and initially they said that:

"Our team had enabled a security feature on the docker daemons to isolate containers with a user namespace.This update may have affected the build that you're experiencing now.For more info about the update: https://docs.docker.com/engine/security/userns-remap/"

Afterwards they said that they're still investigating.

I doubt that this is the issue. I am confident it's due to storage type "overlay" and/or d_type support:

https://stackoverflow.com/questions/42183994/dockerfile-rm-rf-fail

https://github.com/moby/moby/issues/31321

Still waiting by them.

Thanks for the workaround suggestion!

Philip Hodder

EDIT: This is scoped to any deletion. Not just symlinks.

Hello,

The Pipelines team has done some investigation into this, and this issue seems to be happening when the following three conditions are met:

Your account has user namespace remapping enabled. This is currently enabled globally. (This was a security feature we've wanted to enable for a while and significantly improves the security of our build cluster.)
You are trying to delete a file.
You are doing the deletion inside of Docker. (With a few additional conditions)

We're still looking for a long term fix for this, but in the meantime you can try the following workarounds:

Contact Atlassian Support. We'll add you to the whitelist to disable user namespace remapping (please reference this Community question so we can verify you fall into these known conditions).
Delete the directory or file in the same Docker build instruction as it's created.

An example of the Docker build workaround. Is starting with this (failing) Docker image:

FROM ubuntu:18.04
RUN echo "test" > file1.txt

# Shows file1.txt
RUN ls -l 

# Attempt to delete file1.txt
RUN rm file1.txt

# Shows that file1.txt still exists. When it should have been deleted.
RUN ls -l

You can then change it to instead be:

FROM ubuntu:18.04
RUN echo "test" > file1.txt \
  && ls -l \
  && rm file1.txt \
  && ls -l

Sorry for the inconvenience. We're still pinpointing the root cause of this.

@Tania Petsouka, your account has been whitelisted (as mentioned in your support ticket). You should be able to run pipelines again without any other workarounds.

Thanks,

Phil

Tania Petsouka

Cool, thanks!

Philip Hodder

Hello,

Thanks,

Phil

Malte Hübner

@Philip Hodder We are experiencing the same issue since the beginning of October.

Our build is based on the official docker image tomcat:8.5-jre8. Our Dockerfile removes all tomcat default apps with this command:

RUN rm -rf /usr/local/tomcat/webapps/

The build runs fine but the final docker image contains all default webapps which should have been deleted. This worked until the beginning of October.

I also think it could be related to d_type OverlayFS-Driver as supposed by @Tania Petsouka:

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Docker will not support running without d_type support in future releases.

Malte Hübner

@Tania Petsouka what exactly solved your problems? Creating the ticket for Atlassian support to add your account to whitelist or creating a new base image?

Tania Petsouka

Hi Malte,

yes that's what we did.

But according to Philip's post above (on Tuesday) i guessed that they found a more global solution. Not sure though.

I'd suggest that you open a ticket as well and add a link to this thread.

Philip Hodder

Hi @Malte Hübner,

Can you open a support ticket for the team to investigate in more detail. Please ask the Support Engineer to ping me on Stride. So I can whitelist your account and see if that helps, before I go to sleep (it's currently 11pm here). I'll likely be offline in an hour.

I'm struggling to reproduce this problem in Pipelines. I used a fairly trivial Dockerfile to try and reproduce this.

FROM tomcat:8.5-jre8
RUN ls -l # Shows webapps directory
RUN rm -rf /usr/local/tomcat/webapps && ls -l # Shows webapps directory has been deleted
RUN ls -l # webapps directory is still deleted

I tried this with my repo both enabled, and disabled user namespace remapping.

Is there a Dockerfile you can provide that will reproduce the problem?

In regards to overlay and d_type support, if you run 'docker info' inside of Pipelines you'll see we're using overlay2 with d_type support.

<span> <br>  Backing Filesystem: extfs</span><span> <br>  Supports d_type: true</span><span> <br>  Native Overlay Diff: true</span><span>Storage Driver: overlay2</span>

None of the referenced issues seem to address overlay2, only overlay. So I'm unsure if that's the issue. Particularly due to the timing of these issues being raised. The only Docker-related change we made at the start of October (that I'm aware of) is the user namespace remapping being enabled.

If needed, I'll chase this up with the rest of the team tomorrow.

Thanks,

Phil

Malte Hübner

Thank you for your reply @Philip Hodder! I just tested the pipeline build and the problem seems to be gone. Your fix from Tuesday fixes my issue. Thanks again!

Philip Hodder

Ah! Brilliant! No problem. Glad to hear it's resolved.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket

bitbucket-pipelines

docker

container

file-system