It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

rm -rf inside container via Bitbucket pipeline

Hi,

we use bitbucket pipelines for our CD to Google cloud.

Among others, we build an apache image. Inside it's dockefile we have a step to remove default apache's vhosts:

RUN rm -rf /etc/apache2/sites-available/*

This worked fine until yesterday. Today the rm stopped working and after searching we found out that it's probably due to bitbucket's filesystem which is formatted without d_type support.

See here http://www.rubyrube.com/2017/06/28/dockerfile-run-rm-not-working-ex-run-rm-rf-usrlocaltomcatwebappsroot/

*We deployed the image outside the pipeline properly, so we're sure that it's related to bitbucket.

Can you advice a workaround for this?

 

Thanks

 

 

 

4 answers

1 accepted

0 votes
Answer accepted

Hello,

We've rolled out some changes that have fixed this issue. Can you please try run your docker builds again. Please comment if you're still having issues with deleting/editing files as part of your Docker build.

Thanks,

Phil

EDIT: This is scoped to any deletion. Not just symlinks.

Hello,

The Pipelines team has done some investigation into this, and this issue seems to be happening when the following three conditions are met:

  • Your account has user namespace remapping enabled. This is currently enabled globally. (This was a security feature we've wanted to enable for a while and significantly improves the security of our build cluster.)
  • You are trying to delete a file.
  • You are doing the deletion inside of Docker. (With a few additional conditions)

We're still looking for a long term fix for this, but in the meantime you can try the following workarounds:

  • Contact Atlassian Support. We'll add you to the whitelist to disable user namespace remapping (please reference this Community question so we can verify you fall into these known conditions).
  • Delete the directory or file in the same Docker build instruction as it's created.

An example of the Docker build workaround. Is starting with this (failing) Docker image:

FROM ubuntu:18.04
RUN echo "test" > file1.txt

# Shows file1.txt
RUN ls -l

# Attempt to delete file1.txt
RUN rm file1.txt

# Shows that file1.txt still exists. When it should have been deleted.
RUN ls -l

You can then change it to instead be:

FROM ubuntu:18.04
RUN echo "test" > file1.txt \
&& ls -l \
&& rm file1.txt \
&& ls -l

Sorry for the inconvenience. We're still pinpointing the root cause of this.

@Tania Petsouka, your account has been whitelisted (as mentioned in your support ticket). You should be able to run pipelines again without any other workarounds.

Thanks,

Phil

Hi Tania,

Do you have any news on that topic?

We experience the same issue. I already created a ticket. 

As a workaround, you could use a customized base docker image, where you already deleted the vhosts.

 

br

Tobi

Hi Tobias,

i sent a ticket as well and initially they said that:

"Our team had enabled a security feature on the docker daemons to isolate containers with a user namespace.This update may have affected the build that you're experiencing now.For more info about the update: https://docs.docker.com/engine/security/userns-remap/"

 Afterwards they said that they're still investigating.

I doubt that this is the issue. I am confident it's due to storage type "overlay" and/or d_type support:

https://stackoverflow.com/questions/42183994/dockerfile-rm-rf-fail

https://github.com/moby/moby/issues/31321

 

Still waiting by them. 

Thanks for the workaround suggestion!

@Tania Petsouka what exactly solved your problems? Creating the ticket for Atlassian support to add your account to whitelist or creating a new base image?

Hi Malte,

yes that's what we did. 

But according to Philip's post above (on Tuesday) i guessed that they found a more global solution. Not sure though.

I'd suggest that you open a ticket as well and add a link to this thread.

Indeed the problem is solved with the fix from Tuesday. I did not run another build after Tuesday. My fault! Thanks for your reply!

@Philip Hodder We are experiencing the same issue since the beginning of October.

Our build is based on the official docker image tomcat:8.5-jre8. Our Dockerfile removes all tomcat default apps with this command:

RUN rm -rf /usr/local/tomcat/webapps/

The build runs fine but the final docker image contains all default webapps which should have been deleted. This worked until the beginning of October. 

I also think it could be related to d_type OverlayFS-Driver as supposed by @Tania Petsouka:  

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Docker will not support running without d_type support in future releases.

Hi @Malte Hübner,

Can you open a support ticket for the team to investigate in more detail. Please ask the Support Engineer to ping me on Stride. So I can whitelist your account and see if that helps, before I go to sleep (it's currently 11pm here). I'll likely be offline in an hour.

I'm struggling to reproduce this problem in Pipelines. I used a fairly trivial Dockerfile to try and reproduce this.

FROM tomcat:8.5-jre8
RUN ls -l # Shows webapps directory
RUN rm -rf /usr/local/tomcat/webapps && ls -l # Shows webapps directory has been deleted
RUN ls -l # webapps directory is still deleted

I tried this with my repo both enabled, and disabled user namespace remapping.

Is there a Dockerfile you can provide that will reproduce the problem?

In regards to overlay and d_type support, if you run 'docker info' inside of Pipelines you'll see we're using overlay2 with d_type support.

Storage Driver: overlay2 
Backing Filesystem: extfs

Supports d_type: true

Native Overlay Diff: true

None of the referenced issues seem to address overlay2, only overlay. So I'm unsure if that's the issue. Particularly due to the timing of these issues being raised. The only Docker-related change we made at the start of October (that I'm aware of) is the user namespace remapping being enabled.

If needed, I'll chase this up with the rest of the team tomorrow.

Thanks,

Phil

Thank you for your reply @Philip Hodder! I just tested the pipeline build and the problem seems to be gone. Your fix from Tuesday fixes my issue. Thanks again!

Ah! Brilliant! No problem. :) Glad to hear it's resolved.

Suggest an answer

Log in or Sign up to answer
This widget could not be displayed.
This widget could not be displayed.
Community showcase
Published Mar 14, 2019 in Bitbucket Pipelines

Building a Bitbucket Pipe as a casual coder

...ipe.sh :  #!/bin/bash source "$(dirname "$0")/common.sh" enable_debug extra_args="" if [[ "${DEBUG}" == "true" ]]; then extra_args="--verbose" fi # mandatory variables R...

985 views 1 16
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you