Bamboo + docker = Could not remove working directory for plan

[dylan_millikin]

Hey guys,

I'm running a docker container as a docker task in order to run tests against a branch of code. The docker container simply runs while mounting the working dir as /data (default) then generates artifacts in that mounted dir that bamboo access directly (thanks to the mount). This works great. The problem comes when bamboo tries to erase the working dir it says it does not have the correct permissions :

19-Sep-2016 10:40:28
Finished publishing of artifact Job artifact: [JUnit], pattern: [**/junit.xml] anchored at: [build/logs] in 1.408 ms

19-Sep-2016 10:40:28
Running post build plugin 'npm Cache Cleanup'

19-Sep-2016 10:40:28
Running post build plugin 'Clover Results Collector'

19-Sep-2016 10:40:28
Running post build plugin 'Docker Container Cleanup'

19-Sep-2016 10:40:28
Could not remove working directory for plan 'REXM-DEV6-UT': /var/atlassian/application-data/bamboo/xml-data/build-dir/REXM-DEV6-UT/build/logs: Operation not permitted

19-Sep-2016 10:40:28
Finalising the build...

Anyone know what could be the cause of this? I'm guessing it's somehow related to the volumes not being removed or something? 

Any help would be appreciated.

Cheers.

[Sergey Podobry _Stellarity Software_ · Accepted solution]

The big problem of docker is that a root in a container is a root on the host. So any files created in docker is owned by root and cannot be removed by a bamboo agent if it is run not as root. Solution: run chown -R $UID:$UID.

[dylan_millikin]

Thanks for the info. In theory if the files in the container were generated with different permissions would that bleed out into the host? Say either have a user in the container with the same user name as bamboo. Or perhaps 777 rights on generated files?

[Sergey Podobry _Stellarity Software_]

Yes, you can chmod to 777 also.

[Manasa Tumu]

I have a similar issue, please let me know how can we resolve this.

Caused by: com.atlassian.bamboo.repository.RepositoryException: Unable to clean source directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/TEST-SC' /home/bamboo/bamboo-agent-home/xml-data/build-dir/TEST-SC/test Operation not permitted
 at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.cleanWorkingDirIfNeeded(VcsCheckoutTask.java:264) at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.access$100(VcsCheckoutTask.java:54)

we are using bamboo 6.5 and this is happening intermittently. 

[jvelapol]

Newer versions of Docker appear to not be so "rigid" in their UID/GID ownership of volumes.  On my local mac (I know it's a different OS), for example, Docker remaps the UID back to my userid inside a volume mapped container, so I don't have to worry about things like that.  I'm running 18.06.1 locally. 

(BEGIN EDIT:) Just checked on a bare centos VM with docker-ce installed.  Something else in the Docker for Mac is doing the UID translation, it's not built in to "Docker version 18.06.1-ce, build e68fc7a". (END EDIT)

Given that, however, our RHEL7 based Bamboo servers are running the latest RHEL-supplied docker version (1.13.x?), we do have plenty of "Final" tasks that do something like:

my_id=$(id -u)
docker run --rm -v $(pwd):/data my_image sh -c "find /data ! -user $my_id -exec chown $my_id {} \;"

Where the my_image is an image that runs as root that we created from scratch, though you could use something like a base OS image like centos, debian, alpine, etc, if you trust dockerhub.

Note that I don't really like the Docker integration in Bamboo, because of reasons like this.  It's not quite as good as it should be.

[MarcelW]

I solved this problem a little differently.

One of my build steps was to create the local folder inside build-dir/MY-JOB-HERE/  using a bash script task. This occurs before the task of building the docker container.

#!/bin/bash
# Make a test-reports/ directory to house the xml test reports
install -d -m 0770 -o bamboo -g bamboo ${bamboo.build.working.directory}/test-reports/
chmod g+s ${bamboo.build.working.directory}/test-reports/

In my docker compose yml file, I have this:

services:
    sel:
        build:
            context: ../
            dockerfile: ./compose/Dockerfile
        image: my_regression
        volumes:
            # this yml file is one folder deep from the root job folder
            - ../test-reports:/logs

This allows bamboo to do the cleanup without having to resort to changing users or permissions INSIDE the docker container.  

Lastly, if you want the files created in the test-reports folder to retain the same permissions as the folder, you could add something like the following to the bash script task.

setfacl -d -m u::rwX,g::rwX,o::- ${bamboo.build.working.directory}/test-reports/

[jvelapol]

Our solution was a bit different, and more of a hack:

Let's say the uid and gid of the bamboo user on the agent host (not the container!) is 1000.  We have our own internal docker registry from which we pull images from to generate containers from.  In there, one of the statements in the Dockerfile looks like:

...
RUN groupadd -g 1000 bamboo
RUN useradd --gid bamboo --create-home --uid 1000 bamboo
...
USER bamboo

Then, when the container runs, it runs as the image's "bamboo" user (which is really run as UID 1000), which matches the UID of the "bamboo" user that's running the agent itself.

Things get a little bit sticky if you have a lot of agent hosts, though - you'll need some way to manage that the UID of the bamboo user is "always" the same on each host.  Otherwise, that breaks down.

That having been said, this is probably the wrong approach - You should probably be mounting RO the local working directory (xml-data/build-dir/XXX), assuming that you need to copy something into there, like the source code, then, as part of the build, copy those contents to another directory on the container only, run the build from there, and make extensive use of the "docker copy" command when done.  You would then flag artifacts that you care about for the build, and specifically "docker copy" those artifacts back to the agent, then remove the docker container.  This assumes, of course, that you're not deploying the container itself as an artifact (we don't).

[marcelo freitas]

Está bem

[marcelo freitas]

show