we use bitbucket pipelines for our CD to Google cloud.
Among others, we build an apache image. Inside it's dockefile we have a step to remove default apache's vhosts:
RUN rm -rf /etc/apache2/sites-available/*
This worked fine until yesterday. Today the rm stopped working and after searching we found out that it's probably due to bitbucket's filesystem which is formatted without d_type support.
*We deployed the image outside the pipeline properly, so we're sure that it's related to bitbucket.
Can you advice a workaround for this?
EDIT: This is scoped to any deletion. Not just symlinks.
The Pipelines team has done some investigation into this, and this issue seems to be happening when the following three conditions are met:
We're still looking for a long term fix for this, but in the meantime you can try the following workarounds:
An example of the Docker build workaround. Is starting with this (failing) Docker image:
RUN echo "test" > file1.txt
# Shows file1.txt
RUN ls -l
# Attempt to delete file1.txt
RUN rm file1.txt
# Shows that file1.txt still exists. When it should have been deleted.
RUN ls -l
You can then change it to instead be:
RUN echo "test" > file1.txt \
&& ls -l \
&& rm file1.txt \
&& ls -l
Sorry for the inconvenience. We're still pinpointing the root cause of this.
@Tania Petsouka, your account has been whitelisted (as mentioned in your support ticket). You should be able to run pipelines again without any other workarounds.
i sent a ticket as well and initially they said that:
"Our team had enabled a security feature on the docker daemons to isolate containers with a user namespace.This update may have affected the build that you're experiencing now.For more info about the update: https://docs.docker.com/engine/security/userns-remap/"
Afterwards they said that they're still investigating.
I doubt that this is the issue. I am confident it's due to storage type "overlay" and/or d_type support:
Still waiting by them.
Thanks for the workaround suggestion!
@Philip Hodder We are experiencing the same issue since the beginning of October.
Our build is based on the official docker image tomcat:8.5-jre8. Our Dockerfile removes all tomcat default apps with this command:
RUN rm -rf /usr/local/tomcat/webapps/
The build runs fine but the final docker image contains all default webapps which should have been deleted. This worked until the beginning of October.
I also think it could be related to d_type OverlayFS-Driver as supposed by @Tania Petsouka:
Docker will not support running without d_type support in future releases.
Hi @Malte Hübner,
Can you open a support ticket for the team to investigate in more detail. Please ask the Support Engineer to ping me on Stride. So I can whitelist your account and see if that helps, before I go to sleep (it's currently 11pm here). I'll likely be offline in an hour.
I'm struggling to reproduce this problem in Pipelines. I used a fairly trivial Dockerfile to try and reproduce this.
RUN ls -l # Shows webapps directory
RUN rm -rf /usr/local/tomcat/webapps && ls -l # Shows webapps directory has been deleted
RUN ls -l # webapps directory is still deleted
I tried this with my repo both enabled, and disabled user namespace remapping.
Is there a Dockerfile you can provide that will reproduce the problem?
In regards to overlay and d_type support, if you run 'docker info' inside of Pipelines you'll see we're using overlay2 with d_type support.
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
None of the referenced issues seem to address overlay2, only overlay. So I'm unsure if that's the issue. Particularly due to the timing of these issues being raised. The only Docker-related change we made at the start of October (that I'm aware of) is the user namespace remapping being enabled.
If needed, I'll chase this up with the rest of the team tomorrow.
Hello! My name is Mark Askew and I am a Premier Support Engineer for products Bitbucket Server/Data Center, Fisheye & Crucible. Today, I want to bring the discussion that Jennifer, Matt, and ...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs