Community
Products
Bamboo
Questions
Why Job -> other -> "Clean working directory after each build" cleans ALL my docker mounted paths?

Why Job -> other -> "Clean working directory after each build" cleans ALL my docker mounted paths?

TL;DR; How to I exclude directory (mounted by docker if this is of any relevance and not subdirectory of: ${bamboo.build.working.directory}) from being cleaned when "Clean working directory after each build" is checked.

Hi,

I have a big project (1,2 mln lines of C++ code) that I obviously require pull request in order to add code to develop and main branches.

One of (obvious) requirements is that code compiles. Compilation is happening inside docker container with following mounts (not my idea, the compilation script has some of those hardcoded and I can not do anything about it):

${bamboo.build.working.directory} -> /opt/src

${bamboo.build.working.directory}/results -> /opt/results

In order to speed up the compilation we tested usage of ccache and it works great. Compilation time down from around 40 minutes to around 4 minutes.

So the idea was to mount:

/home/ccache -> /home/ccache

so that each docker will have access to that folder and cache will be shared.

Everything works OK as long as option "Clean working directory after each build" for the build job is unchecked.
Now we want this option checked and ${bamboo.build.working.directory} cleaned after the build since we are talking 2+GB. Given all the other builds and branches ... there is a limited NVMe storage for the builders ...

My understanding was that "Clean working directory after each build" will clean only ${bamboo.build.working.directory} not ALL directories mounted by docker.

I really need to exclude /home/ccache from the clean.

How do I do that?

1 answer

1 accepted

1 vote

Answer accepted

Hello @Mariusz Dullak

Thank you for the detailed explanation.

We were able to reproduce this issue and for that have opened a new bug:

BAM-22118 - Bamboo Docker Runner Job will not substitute ${bamboo.working.directory} variable if that is customised as a mount volume option

We recommend you watch it to be notified of any changes in the future.

There are a number of factors that determine how Atlassian prioritizes bug fixes. You can learn more about this by reading our Atlassian Data Center and Server Bug Fix Policy. And you can visit our dashboard if you would like to search for any existing bugs in the future.

In the bug have included a workaround that you can use. We'll give more details below:

Do not use nested volumes such as:

${bamboo.build.working.directory} >> ${bamboo.build.working.directory}
${bamboo.build.working.directory}/results >> /opt/results

Using nested volumes in Bamboo has the same effect as in Docker, when the nested source location doesn't exist (a fresh build directory), Docker will try to create it and assign it to the volume, but that doesn't work quite well as it will result in a broken git checkout as Git will consider the ${bamboo.build.working.directory} as not empty, Bamboo will then try to delete its content to make Git happy and that will break any volume assignments.

In your case we recommend you use only:

${bamboo.build.working.directory} >> ${bamboo.build.working.directory}
${bamboo.tmp.directory} >> ${bamboo.tmp.directory}
/home/ccache >> /home/ccache

Then add a Script task that will create symlinks to /opt/src and /opt/results:

$ ln -sf ${bamboo_build_working_directory} /opt/src
$ ln -sf ${bamboo_build_working_directory}/results /opt/results 
$ cd /opt/src && run-build-script

Cheers,

Eduardo Alvarenga
Atlassian Support APAC

--please don't forget to Accept the answer if the reply is helpful--

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Eduardo Alvarenga

thank you for your help.

Unfortunately is seems that you focused on the issue that I am working around with simple

cd /opt/src

(please see my comments in the bug you mentioned) and is not the real issue.

Let me try to restate the issue.

Steps to reproduce:

use Docker in bamboo job.

Mount

${bamboo.build.working.directory} -> /opt/src
/home/ccache -> /home/ccache

Check option to "Clean working directory after each build" in the job other section.

In the job put some files to

/home/ccache

as simple test

touch /home/ccache/test

Actual result:

on bamboo agent host following folders are cleaned:

${bamboo.build.working.directory}
/home/ccache

Expected result:

on bamboo agent host only following folder is cleaned:

${bamboo.build.working.directory}

this is what the options says: working directory. Not all folders mounted in docker.

The file test in

/home/ccache/test

should remain on the host and next job in the plan (or other branches of the plan or even other plans) can access it mounting same folder in its docker.

I hope that it is much clearer now what the actual issue is. Please do note hesitate to contact me for more details

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hello @Mariusz Dullak

Thank you for the further explanation.

I cannot reproduce it locally. In my local instance /home/ccache content remains intact and any generated files are still there after the Plan is finished with an enabled Clean working directory after each build option.

Can you provide us with the exact version of Bamboo you are running along with an exported Plan in Java Specs format?

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hey @Eduardo Alvarenga

So I did some more testing and apparently that has something to do with me mounting

${bamboo.build.working.directory} -> /opt/src

Because when I have this mounted the file disapears.

When I mount:

${bamboo.build.working.directory} -> ${bamboo.build.working.directory}

the file stays as expected.

So it looks like there is a problem with the docker mount.

Here are details of my "failing" plan (sorry for the formatting but apparently copy-paste does not work well here).

What I do is I run single scrip task in shell with content:

#!/bin/bash

touch /home/ccache/test
sleep 10
echo "test" >> /home/ccache/test
sleep 10

exit 0

bamboo version is: 9.2.1 build 90201. Here is JavaSpec

import com.atlassian.bamboo.specs.api.BambooSpec;
import com.atlassian.bamboo.specs.api.builders.BambooKey;
import com.atlassian.bamboo.specs.api.builders.BambooOid;
import com.atlassian.bamboo.specs.api.builders.Variable;
import com.atlassian.bamboo.specs.api.builders.docker.DockerConfiguration;
import com.atlassian.bamboo.specs.api.builders.permission.PermissionType;
import com.atlassian.bamboo.specs.api.builders.permission.Permissions;
import com.atlassian.bamboo.specs.api.builders.permission.PlanPermissions;
import com.atlassian.bamboo.specs.api.builders.plan.Job;
import com.atlassian.bamboo.specs.api.builders.plan.Plan;
import com.atlassian.bamboo.specs.api.builders.plan.PlanIdentifier;
import com.atlassian.bamboo.specs.api.builders.plan.Stage;
import com.atlassian.bamboo.specs.api.builders.plan.branches.BranchCleanup;
import com.atlassian.bamboo.specs.api.builders.plan.branches.PlanBranchManagement;
import com.atlassian.bamboo.specs.api.builders.plan.configuration.AllOtherPluginsConfiguration;
import com.atlassian.bamboo.specs.api.builders.plan.configuration.ConcurrentBuilds;
import com.atlassian.bamboo.specs.api.builders.project.Project;
import com.atlassian.bamboo.specs.builders.task.ScriptTask;
import com.atlassian.bamboo.specs.builders.trigger.BitbucketServerTrigger;
import com.atlassian.bamboo.specs.util.BambooServer;
import com.atlassian.bamboo.specs.util.MapBuilder;

@BambooSpec
public class PlanSpec {

public Plan plan() {
final Plan plan = new Plan(new Project()
.oid(new BambooOid("14g9zf7pyybcy"))
.key(new BambooKey("ATV"))
.name("Atlassian Toolchain Verification")
.description("Project for testing purposes. Lead Mariusz Dullak."),
"ccache test",
new BambooKey("CT"))
.oid(new BambooOid("14g0a7mcr4xz7"))
.pluginConfigurations(new ConcurrentBuilds())
.stages(new Stage("Default Stage")
.jobs(new Job("Default Job",
new BambooKey("JOB1"))
.pluginConfigurations(new AllOtherPluginsConfiguration()
.configuration(new MapBuilder()
.put("custom", new MapBuilder()
.put("auto", new MapBuilder()
.put("label", "")
.put("regex", "")
.build())
.put("clover.path", "")
.put("buildHangingConfig.enabled", "false")
.put("ncover.path", "")
.build())
.build()))
.tasks(new ScriptTask()
.description("Compile")
.inlineBody("#!/bin/bash\n\ntouch /home/ccache/test\nsleep 10\necho \"test\" >> /home/ccache/test\nsleep 10\n\nexit 0"))
.cleanWorkingDirectory(true)
.dockerConfiguration(new DockerConfiguration()
.image("debian:11-slim")
.withoutDefaultVolumes()
.volume("${bamboo.working.directory}", "/opt/src")
.volume("${bamboo.tmp.directory}", "${bamboo.tmp.directory}")
.volume("/home/ccache", "/home/ccache"))))
.linkedRepositories("Nova core U20.04")

.triggers(new BitbucketServerTrigger())
.variables(new Variable("workers",
"48"))
.planBranchManagement(new PlanBranchManagement()
.delete(new BranchCleanup())
.notificationForCommitters());
return plan;
}

public PlanPermissions planPermission() {
final PlanPermissions planPermission = new PlanPermissions(new PlanIdentifier("ATV", "CT"))
.permissions(new Permissions()
.userPermissions("dullak", PermissionType.EDIT, PermissionType.VIEW, PermissionType.ADMIN, PermissionType.CLONE, PermissionType.BUILD));
return planPermission;
}

public static void main(String... argv) {
//By default credentials are read from the '.credentials' file.
BambooServer bambooServer = new BambooServer("https://bamboo.espera.de");
final PlanSpec planSpec = new PlanSpec();

final Plan plan = planSpec.plan();
bambooServer.publish(plan);

final PlanPermissions planPermission = planSpec.planPermission();
bambooServer.publish(planPermission);
}
}

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hello @Mariusz Dullak

Thanks for the extra details. I appreciate that.

We have dug into it a little bit and come up with more findings! That ended up on another bug that is related to the first one:

BAM-22120 - Bamboo will delete the whole Docker Runner container including long-term mounted volumes data(!) if the ${bamboo.build.working.directory} doesn't exist or can't be accessed

As usual, check the link, place comments and follow it to be notified of changes moving forward. The workaround is pretty much the same as the first bug, do not use a custom ${bamboo.build.working.directory} directory.

Thanks a lot for reporting that! We really appreciate the collaboration and understanding.

Sincerely,

Eduardo Alvarenga
Atlassian Support APAC

--please don't forget to Accept the answer if the reply is helpful--

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Like • Steffen Opel _Utoolity_ likes this

Hi @Eduardo Alvarenga

thank you for taking your time, confirming the issue, finding that the clean script is executed on / (root) and reporting all of this as a bug.

I believe that my question is now answered. This is a bug. Hopefully an easy one to fix and it will be fixed soon.

In the meantime, let me implement workarounds in all of my plans impacted.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Like • Eduardo Alvarenga likes this

Suggest an answer

Was this helpful?

Thanks!

Bamboo

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Why Job -> other -> "Clean working directory after each build" cleans ALL my docker mounted paths?

1 answer

1 accepted

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events

Ask a question

Start a discussion

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Why Job -> other -> "Clean working directory after each build" cleans ALL my docker mounted paths?

1 answer

1 accepted

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events