Bamboo + docker = Could not remove working directory for plan

Hey guys,

I'm running a docker container as a docker task in order to run tests against a branch of code. The docker container simply runs while mounting the working dir as /data (default) then generates artifacts in that mounted dir that bamboo access directly (thanks to the mount). This works great. The problem comes when bamboo tries to erase the working dir it says it does not have the correct permissions :

19-Sep-2016 10:40:28
Finished publishing of artifact Job artifact: [JUnit], pattern: [**/junit.xml] anchored at: [build/logs] in 1.408 ms

19-Sep-2016 10:40:28
Running post build plugin 'npm Cache Cleanup'

19-Sep-2016 10:40:28
Running post build plugin 'Clover Results Collector'

19-Sep-2016 10:40:28
Running post build plugin 'Docker Container Cleanup'

19-Sep-2016 10:40:28
Could not remove working directory for plan 'REXM-DEV6-UT': /var/atlassian/application-data/bamboo/xml-data/build-dir/REXM-DEV6-UT/build/logs: Operation not permitted

19-Sep-2016 10:40:28
Finalising the build...

Anyone know what could be the cause of this? I'm guessing it's somehow related to the volumes not being removed or something? 

 

Any help would be appreciated.

Cheers.

4 answers

1 accepted

The big problem of docker is that a root in a container is a root on the host. So any files created in docker is owned by root and cannot be removed by a bamboo agent if it is run not as root. Solution: run chown -R $UID:$UID.

Thanks for the info. In theory if the files in the container were generated with different permissions would that bleed out into the host? Say either have a user in the container with the same user name as bamboo. Or perhaps 777 rights on generated files?

Yes, you can chmod to 777 also.

I have a similar issue, please let me know how can we resolve this.

Caused by: com.atlassian.bamboo.repository.RepositoryException: Unable to clean source directory '/home/bamboo/bamboo-agent-home/xml-data/build-dir/TEST-SC' /home/bamboo/bamboo-agent-home/xml-data/build-dir/TEST-SC/test Operation not permitted
 at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.cleanWorkingDirIfNeeded(VcsCheckoutTask.java:264) at com.atlassian.bamboo.plugins.vcs.task.VcsCheckoutTask.access$100(VcsCheckoutTask.java:54)

 

we are using bamboo 6.5 and this is happening intermittently. 

Newer versions of Docker appear to not be so "rigid" in their UID/GID ownership of volumes.  On my local mac (I know it's a different OS), for example, Docker remaps the UID back to my userid inside a volume mapped container, so I don't have to worry about things like that.  I'm running 18.06.1 locally. 

(BEGIN EDIT:) Just checked on a bare centos VM with docker-ce installed.  Something else in the Docker for Mac is doing the UID translation, it's not built in to "Docker version 18.06.1-ce, build e68fc7a". (END EDIT)

Given that, however, our RHEL7 based Bamboo servers are running the latest RHEL-supplied docker version (1.13.x?), we do have plenty of "Final" tasks that do something like:

my_id=$(id -u)
docker run --rm -v $(pwd):/data my_image sh -c "find /data ! -user $my_id -exec chown $my_id {} \;"

Where the my_image is an image that runs as root that we created from scratch, though you could use something like a base OS image like centos, debian, alpine, etc, if you trust dockerhub.

Note that I don't really like the Docker integration in Bamboo, because of reasons like this.  It's not quite as good as it should be.

Our solution was a bit different, and more of a hack:

Let's say the uid and gid of the bamboo user on the agent host (not the container!) is 1000.  We have our own internal docker registry from which we pull images from to generate containers from.  In there, one of the statements in the Dockerfile looks like:

...
RUN groupadd -g 1000 bamboo
RUN useradd --gid bamboo --create-home --uid 1000 bamboo
...
USER bamboo

Then, when the container runs, it runs as the image's "bamboo" user (which is really run as UID 1000), which matches the UID of the "bamboo" user that's running the agent itself.

Things get a little bit sticky if you have a lot of agent hosts, though - you'll need some way to manage that the UID of the bamboo user is "always" the same on each host.  Otherwise, that breaks down.

That having been said, this is probably the wrong approach - You should probably be mounting RO the local working directory (xml-data/build-dir/XXX), assuming that you need to copy something into there, like the source code, then, as part of the build, copy those contents to another directory on the container only, run the build from there, and make extensive use of the "docker copy" command when done.  You would then flag artifacts that you care about for the build, and specifically "docker copy" those artifacts back to the agent, then remove the docker container.  This assumes, of course, that you're not deploying the container itself as an artifact (we don't).

Suggest an answer

Log in or Sign up to answer
Community showcase
Published yesterday in Jira Service Desk

Frequently Asked Questions for Jira Service Desk (AKA JSD FAQ)

Just the FAQs! Welcome to the Community, and this Focused-FAQ! Here, we've pulled together some of the most frequently asked questions associated with Jira Service Desk (JSD). We hope ...

102 views 0 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you