The Atlassian documentation often frustrates me. It provides relatively clear instructions for the straightforward operations that they expect people to perform, but there's virtually no detailed information about how the system operates beyond that.
For example, I'm trying to understand exactly when when Bamboo (4.4.3 but we expect to upgrade to 5.0 next week) will remove workspaces for builds that it's done. We have auto-branching enabled (using a Git repository) for some of our plans, and we're running out of disk space. I have expiration set for 7 days on these plans since they're for developers' personal builds. The Bamboo server believes that these builds are deleted: there are no logs, artifacts, etc. for these branch plans.
When I go look on the system, I see a large number of ancient workspaces (from March etc.) chewing up lots of space. If my base plan ID is MAIN-BRANCH then these auto-branch workspaces will have names like MAIN-BRANCH1, MAIN-BRANCH24, etc.
Can anyone explain how Bamboo manages these workspaces? When they are deleted, if ever? When they are reused, if ever? Do I have to clean them up myself?
Well that's depressing, but at least I have an answer!
So the next question is, how do I know when I can delete these workspaces? That is, how do I know which workspaces are still needed by Bamboo plans, and which are not?
Is there some facility in Bamboo that would make this easier? Or do I have to completely invent it myself?
It's doable but not simple. You can use https://docs.atlassian.com/bamboo/REST/5.0-SNAPSHOT/#d2e101 to get all plans and then use the branch REST endpoint to get all branches for each plan. Anything you haven't found has to be deleted.
You could also delete everything once per month/fortnight. That's way simpler.
Thanks for the info on the REST interface. Deleting things older than XX days seems like the way to go... unfortunately I need this to work on Linux, MacOS, and Windows Bamboo agents. Adding a cron job to run regularly on Linux and MacOS is no problem but I've no idea how to do something like that on Windows. Off to StackOverflow I guess...
Hi Przemek. I've done more investigation and I think the answer is not so simple as your statement above. I believe you that for "normal" plans, every plan gets a separate workspace directory and these are not deleted.
However, I think that something different happens with automatic branch plans, the ones where the workspace is named PROJECT-PLANXX where XX is some number. It appears to me that these working directories ARE reused. Or to be more specific, the number XX is reassigned to different branches over time by Bamboo.
I've seen multiple cases where a working directory PROJ-PLAN10 is set to one (old) Git branch on one Bamboo agent, but on another Bamboo agent it's set to a different (new) Git branch.
Can you re-check with the Bamboo folks and re-verify (a) whether working directories are re-used during auto-branch specifically, and (b) if they are re-used, what the algorithm is for their re-use? When does Bamboo decide it's OK to re-use them?
Thanks Przemek. However, this kind of begs the question. If Bamboo uses the first available plan key, then my question is how does Bamboo decide when a plan key is available for use? I've looked in the XML files you suggest above but I don't immediately see how this translates into whether the plan key is in use or not.
Yes, I understand that the first available key is taken. But how does Bamboo decide if a key is available or not?
It can't be "the first key that's never been used for anything in Bamboo before", otherwise there would never be any re-use of keys and I've definitely seen re-use.
So Bamboo must be able to look at each key it knows about and make a decision as to whether that key is available, or not. But what criteria or algorithm does it use to make that decision?
OK. By "build table" I assume you mean the list of builds that currently exist (haven't been expired) on the server. So basically what you're saying is that any plan key which has no builds associated with it is "available" to be used by some other (branch of a) plan.
I'm actually happy with this and I hope you do NOT change it, unless you also at the same time (or no later anyway) implement some automated way of cleaning up old workspaces. Reusing plan keys at least keeps the disk space somewhat finite. If every single brance always used a new key we'd never get any re-use of disk space and it would grow indefinitely.
I've only used 4.4.5, 5.0.0, & 5.0.1. I have a single instance running on our own hardware running. With 1 local build agent it always reused the working directory, now running with 15 and they are still reused but seems like only by the local build agent that created it.
There is this option is the job edit.
We use Subversion, but there is even an option to force clean remove a checkout which would also suggest the directory is reused:
Hi Jason. Thanks for your note. However, I'm aware of that option and it's not what we want. That option will clean out the working directory after every build, which means that the next build will take significantly longer even if there's only a small change. Preserving the previous build, at least for us, drastically reduces built time for the second and subsequent builds which is important.
Also, I agree that each plan will create one working directory (per agent that it's built on) and re-use it. However, we are using the auto-branching feature of Bamboo. Perhaps you're not familiar with it, but this allows a given plan to be run with distinct build numbers for each branch that exists in the source repository.
In our environment Bamboo builds are not just the purview of the "release engineering" team creating official package builds; every developer must get a clean build (unit and functional tests are integrated with the build as well) on their private branch before they are allowed to merge into the mainline development branch. Also we don't have enough machines for each developer to have a personal Linux, Mac, and Windows system to themselves so we use the Bamboo build agents to work out portability issues that are not found on the developer's personal machine.
So, we have a LOT of personal developer branches being automatically created in Bamboo, and a small number (usually more than 2 but less than 15 unless it's a big change) of builds per branch. So we don't want to clean the working directory after each build, but we DO want to clean them eventually, when they're no longer needed, otherwise we run out of storage on the build systems and builds start to fail. And of course, forcing a clean build only before each build is really not what we want: it's the worst of all worlds (slow builds AND we still use all the same amount of disk space).
We do the same thing but able/willing to deal with the extra time. We have had problems with folders being moved out to library projects and referenced as an external. If the update goes over multiple revisions it can corrupt the checkout/working copy; this was the easiest (not the fastest) solution to not have to manually find and delete a single working copy.
We've come across an issue on how this "reuse of branches" causes problems with Deployment projects.
- Create a release ("release-PROJ-1") and link to build PROJ build #1 (e.g. Bamboo url http://localhost:8085/browse/PROJ0-1)
- Deploy this
- Then at some point when Bamboo expires this particular branch
- Creating a new branch, and say it happens to use the same url
- When you try to create a release linking to this new branch, Bamboo says:
"No release with the artifacts of this build exists."
Seems the Bamboo release will still reference the url of the old branch even though a new one has been created (which just happens to reuse it).
This has got to be a bug. Has anyone seen this before?
The only short-term workaround I can think of is to have Bamboo NOT expire any of our branches, so that each branch url is distinct. However, since we have over 200 build plans and use branches heavily, that also prevents us from easily cleaning up unreleased branches.
Over the next several weeks we'll be sharing some of our Getting Started guides here in the community. Throughout this series of posts, we'd love to hear from customers and non-customers ab...
Connect with like-minded Atlassian users at free events near you!Find a group
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no AUG chapters near you at the moment.Start an AUG
You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs