Describe exactly when Bamboo workspaces are cleaned/removed?

[Paul Smith]

The Atlassian documentation often frustrates me. It provides relatively clear instructions for the straightforward operations that they expect people to perform, but there's virtually no detailed information about how the system operates beyond that.

For example, I'm trying to understand exactly when when Bamboo (4.4.3 but we expect to upgrade to 5.0 next week) will remove workspaces for builds that it's done. We have auto-branching enabled (using a Git repository) for some of our plans, and we're running out of disk space. I have expiration set for 7 days on these plans since they're for developers' personal builds. The Bamboo server believes that these builds are deleted: there are no logs, artifacts, etc. for these branch plans.

When I go look on the system, I see a large number of ancient workspaces (from March etc.) chewing up lots of space. If my base plan ID is MAIN-BRANCH then these auto-branch workspaces will have names like MAIN-BRANCH1, MAIN-BRANCH24, etc.

Can anyone explain how Bamboo manages these workspaces? When they are deleted, if ever? When they are reused, if ever? Do I have to clean them up myself?

[Przemek Bruski · Accepted solution]

These workspaces are not removed. They are also never reused. Yes, you'll have to clean them up yourself. I've opened an improvement request for this: https://jira.atlassian.com/browse/BAM-13589 .

[Paul Smith]

Well that's depressing, but at least I have an answer!

So the next question is, how do I know when I can delete these workspaces? That is, how do I know which workspaces are still needed by Bamboo plans, and which are not?

Is there some facility in Bamboo that would make this easier? Or do I have to completely invent it myself?

Thanks!

[Przemek Bruski]

It's doable but not simple. You can use https://docs.atlassian.com/bamboo/REST/5.0-SNAPSHOT/#d2e101 to get all plans and then use the branch REST endpoint to get all branches for each plan. Anything you haven't found has to be deleted.

You could also delete everything once per month/fortnight. That's way simpler.

[Jason Monsorno]

What is considered a workspace? How does that relate to a working copy?

[Paul Smith]

I probably don't have the terminology correct, but when I say "workspace" I mean the extracted (in my case Git) repository the Bamboo agent creates on the build server, where the builds are invoked. Maybe that's what you mean by "working copy".

[Paul Smith]

Thanks for the info on the REST interface. Deleting things older than XX days seems like the way to go... unfortunately I need this to work on Linux, MacOS, and Windows Bamboo agents. Adding a cron job to run regularly on Linux and MacOS is no problem but I've no idea how to do something like that on Windows. Off to StackOverflow I guess...

[Paul Smith]

Hi Przemek. I've done more investigation and I think the answer is not so simple as your statement above. I believe you that for "normal" plans, every plan gets a separate workspace directory and these are not deleted.

However, I think that something different happens with automatic branch plans, the ones where the workspace is named PROJECT-PLANXX where XX is some number. It appears to me that these working directories ARE reused. Or to be more specific, the number XX is reassigned to different branches over time by Bamboo.

I've seen multiple cases where a working directory PROJ-PLAN10 is set to one (old) Git branch on one Bamboo agent, but on another Bamboo agent it's set to a different (new) Git branch.

Can you re-check with the Bamboo folks and re-verify (a) whether working directories are re-used during auto-branch specifically, and (b) if they are re-used, what the algorithm is for their re-use? When does Bamboo decide it's OK to re-use them?

[Przemek Bruski]

When Bamboo creates a branch, it will use the first available plan key, which may look like workspace reuse.

Bamboo determines the "owner" of the workspace using the files in bamboo-agent-home/xml-data/build-dir/repositoryData directory.

[Paul Smith]

Thanks Przemek. However, this kind of begs the question. If Bamboo uses the first available plan key, then my question is how does Bamboo decide when a plan key is available for use? I've looked in the XML files you suggest above but I don't immediately see how this translates into whether the plan key is in use or not.

Thanks!

[Przemek Bruski]

Plan key selection is done on the server - basically, the first available key is taken (the key number is suffixed by a number that gets incremented until it's unique).

The XML file contains the plan id.

[Paul Smith]

Yes, I understand that the first available key is taken. But how does Bamboo decide if a key is available or not?

It can't be "the first key that's never been used for anything in Bamboo before", otherwise there would never be any re-use of keys and I've definitely seen re-use.

So Bamboo must be able to look at each key it knows about and make a decision as to whether that key is available, or not. But what criteria or algorithm does it use to make that decision?

[Przemek Bruski]

It checks the 'build' table to see which keys are already taken.

We are planning to change that behaviour to make the keys not reusable, but there's no schedule for that yet.

[Paul Smith]

OK. By "build table" I assume you mean the list of builds that currently exist (haven't been expired) on the server. So basically what you're saying is that any plan key which has no builds associated with it is "available" to be used by some other (branch of a) plan.

I'm actually happy with this and I hope you do NOT change it, unless you also at the same time (or no later anyway) implement some automated way of cleaning up old workspaces. Reusing plan keys at least keeps the disk space somewhat finite. If every single brance always used a new key we'd never get any re-use of disk space and it would grow indefinitely.

[Ron Chan]

We've come across an issue on how this "reuse of branches" causes problems with Deployment projects.

- Create a release ("release-PROJ-1") and link to build PROJ build #1 (e.g. Bamboo url http://localhost:8085/browse/PROJ0-1)
- Deploy this
- Then at some point when Bamboo expires this particular branch
- Creating a new branch, and say it happens to use the same url
- When you try to create a release linking to this new branch, Bamboo says:
"No release with the artifacts of this build exists."

Seems the Bamboo release will still reference the url of the old branch even though a new one has been created (which just happens to reuse it).

This has got to be a bug. Has anyone seen this before?

The only short-term workaround I can think of is to have Bamboo NOT expire any of our branches, so that each branch url is distinct. However, since we have over 200 build plans and use branches heavily, that also prevents us from easily cleaning up unreleased branches.

[Jason Monsorno]

I've only used 4.4.5, 5.0.0, & 5.0.1. I have a single instance running on our own hardware running. With 1 local build agent it always reused the working directory, now running with 15 and they are still reused but seems like only by the local build agent that created it.

There is this option is the job edit.

We use Subversion, but there is even an option to force clean remove a checkout which would also suggest the directory is reused:

[Paul Smith]

Hi Jason. Thanks for your note. However, I'm aware of that option and it's not what we want. That option will clean out the working directory after every build, which means that the next build will take significantly longer even if there's only a small change. Preserving the previous build, at least for us, drastically reduces built time for the second and subsequent builds which is important.

Also, I agree that each plan will create one working directory (per agent that it's built on) and re-use it. However, we are using the auto-branching feature of Bamboo. Perhaps you're not familiar with it, but this allows a given plan to be run with distinct build numbers for each branch that exists in the source repository.

In our environment Bamboo builds are not just the purview of the "release engineering" team creating official package builds; every developer must get a clean build (unit and functional tests are integrated with the build as well) on their private branch before they are allowed to merge into the mainline development branch. Also we don't have enough machines for each developer to have a personal Linux, Mac, and Windows system to themselves so we use the Bamboo build agents to work out portability issues that are not found on the developer's personal machine.

So, we have a LOT of personal developer branches being automatically created in Bamboo, and a small number (usually more than 2 but less than 15 unless it's a big change) of builds per branch. So we don't want to clean the working directory after each build, but we DO want to clean them eventually, when they're no longer needed, otherwise we run out of storage on the build systems and builds start to fail. And of course, forcing a clean build only before each build is really not what we want: it's the worst of all worlds (slow builds AND we still use all the same amount of disk space).

[Jason Monsorno]

We do the same thing but able/willing to deal with the extra time. We have had problems with folders being moved out to library projects and referenced as an external. If the update goes over multiple revisions it can corrupt the checkout/working copy; this was the easiest (not the fastest) solution to not have to manually find and delete a single working copy.

[Paul Smith]

You might consider a more robust SCM solution then :-p. We've never had these sorts of issues with Git.