follow up to the 'server specifications' question: JVM's used by Bamboo, agents, and builds.

[Ken Wood]

Related to "https://answers.atlassian.com/questions/11721/server-specifications", I have these questions:

For discussion sake, assume a linux based computer running, say, Bamboo 3.3.x, with 10 local agents.

1.a Does each local agent run in the Bamboo JVM instance?

1.b Or, does each local agent have it's own JVM instance?

2.a When a local agent is running a build, and the build does a java compilation, is the compilation using the Agent's JVM?

2.b Or, does the java compilation get it's own JVM instance to run in?

[Przemek Bruski · Accepted solution]

1 local agents run in Bamboo's JVM instance. They typically spawn external processes anyway.

2. No, there is no in-process compilation done. All javac calls are done by external processes.

[ReneR]

Oh and yes, we run a single remote agent inside each virtualised instance. We use Dell R510's as host nodes with a mix of OpenVZ and KVM running inside of RHEL5 and Scientific Linux. We can get between 12-16 containers/guests on each host node depending on how we size them. Guest OS is usually Fedora or CentOS which we manage with Puppet.

[Ken Wood]

OK, thanks for the clarification. On the bamboo box we only do compilation and packaging, no tests. We run testNG or junit tests on other VM instances dedicated to that specific test.

[ReneR]

TEN local agents on the same host in addition to the Bamboo server? Your builds must be tiny and not need a lot of memory... Co-locating of agents is not something I like to do as a build engineer, unless I can make absolutely certain all of my builds are well-behaved and tear down every resource they use on the build machine and don't inject anything into the system that could confuse a build following right after it.

[Ken Wood]

That surprises me. Out of the box Bamboo supports multiple local agents, and I don't see anything in the documentation urging caution in relation to multiple agents on the Bamboo box.

I agree builds should not inject anything into the system (ours don't), but we don't remove the working copy after a build finishes.... in case there is an issue with the build, developers want to be able to examine the working copy.

As to memory, if th

[ReneR]

The issue is giving each individual build as pristine an environment as you can manage. An agent is not a 'container', virtual instance or anything similar. You can run as many 'agents' as your hardware can handle. But all those processes are going to be contending for resources and in many cases re-using things like the shared system environment variables, credentials (unless you run each agent with a different userid). Think of 10 developers sharing the same system and running their own builds at the same time, with or without the benefit of separate userids. So is not an issue for Bamboo but rather just good practices and designing your build environment for reliability and predictability

[Ken Wood]

Ok, but I'd have the same problem if I ran multiple remote agents on a different box than the Bamboo server. I.e. the agents would be contending for resources. We run between 90 and 130 builds/day, and of course they are not uniformily distributed. Sometimes nothing is happening, and at other times we have 7 or 8 builds running at once. We don't have the budget to have 10 or 15 linux boxes dedicated to builds.

But, to make sure I have left no stone unturned, what is your feeling about 10 or 15 virtual instances, each with only one remote agent? That would isolate each build's environment.

[ReneR]

Are you talking about performance? Because I'm not. When I talk about 'contention for resources' I mean starting multiple versions of your app or a web browser against which you're running Unit or Functional tests. Open filehandles, ports, displays, databases, stuff like that.

We use heavy use of virtualisation at Atlassian because most of our build runs need to start up their own app instances (plus a web browser) to run their tests. It would be very difficult to keep things sane if you wanted to start a bunch of those on the same host without some sort of isolation. Not to mention they all want to stick test data into a database running locally.

To go even a step further, we use Amazon EC2 a lot too which spins up a brand new instance for every single build. Doesn't get cleaner than that.

So if you can virtualize, I'd go with that. But again that's my opinion as a build engineer and as someone who's chased flakey builds and found it was due to the environment being left in a 'dirty' state by a previous build. Red builds due to environment flakiness is a headache you don't want :)

If your concern is optimizing the level of hardware you have dedicated to builds so that you always have exactly as much as you need, the moment you need it and nothing more then look at integrating with EC2.

If your builds are not very intrusive, you should be fine running all the agents on the same box. Performance issues aside, what I was trying to warn you about was the possibility of chasing build failures that are due to an unstable environment rather than a bonafide failure.