new vm or docker container for every build

Question

I am looking for a way to have each build run in complete isolation. I also need for each build to start with a "clean" environment. Conceptually, I want it to work like elastic bamboo with one difference... When the remote agent goes to an idle state, I want the remote VM to be destroyed not just shutdown. When a new build is queued, it should create a brand new VM for the build. This allows for build plans to install system level and/or make changes to the system configuration without the risk of impacting other builds.

We don't use EC2 but we do have a private cloud using VMWare's vCloud director. There do not appear to be any plugins currently available that work with vCloud director. I am considering writing a plugin for Bamboo that could provide this type of functionality but I think that it is going to be a fairly high level of effort (if possible at all).

I am wondering if i might be able to accomplish this using Docker. I have already tested out the docker support and 5.7 and it looks very promising. Adding the "--rm=true" parameter to the command line used to run the image causes the container to be removed when the process exits. When it is restarted, it is a brand new environment.

Unfortunately, I have not been able to come up with a way to have the remote agent exit after a build completes so that the docker container can stop running.  If I could figure that part out then I could could wrap it in another script to restart it.

However, even if I solve that problem, this approach causes a new remote agent to be registered on the bamboo server every time the docker container starts.  Is there a way to get around that issue?

Jacobus van Hee · Answer

I am trying to do something similar. At the beginning of the plan I want to start a docker container (using atlassian/bamboo-java-agent:latest). The container includes the bamboo remote agent so as soon as the container is started it registers a Remote Agent with the bamboo server (agent name and container UUID is the same). I now want the rest of the build plan to execute using that specific agent and that way the code will be deployed and tested on that specific container.

StannousBaratheon · Answer

We're working on a Docker plugin for Bamboo 5.8 which should help with your scenario. The plugin will provide 3 Bamboo build tasks to build, run and push Docker images. The Docker run task will run with the --rm=true parameter by default but you can also optionally run the container detached (--detach) and Bamboo will remove the detached containers at the end of the job even if the container is still running.

These tasks run on existing agents with the Docker capability (not inside a Docker Bamboo agent) so there's no need to terminate the remote agent.

Caley Goff · Answer

How do I come up with a way to have the remote agent exit after a build completes so that the docker container can stop running? Adding the "--rm=true" parameter to the command line used to run the image causes the container to be removed when the process exits. When it is restarted, it is a brand new environment.

While I do not know if there will ever be a plugin in the interim there are two different things you can do. We are exploring similar options ourselves.

* You can assign executables to agents or even Bamboo itself. Think of it as giving Bamboo the ability or skillset to do something. Anyway you can do this under the administration panel of Bamboo (steps at the bottom).
* Simply use the script task with inline script. Write out your docker steps there as you would if you were writing a bash script or executing the commands in a shell session.

Both above examples will allow you to add that parameter to your docker command. You can then run whatever commands the user running the bamboo instance (or agent) thats being used can on the system.

If you choose to add the docker command as an executable you can literally add a final command task to the plan. You would choose the docker executable from the drop down (see below to get docker to appear there) and add  "--rm=true"  to the argument box.

Note: I recommend you add this task (command or script) as a final task. This way no matter what happens in the build pipeline, Bamboo will always execute that task and exit the docker builder and/or destroy the environment.

However, even if I solve that problem, this approach causes a new remote agent to be registered on the bamboo server every time the docker container starts.  Is there a way to get around that issue?

I honestly found a way around this by reusing the UUID when replicating agents. I suppose it would be acceptable to reuse the UUID for each project agent that lives in an environment. This is something I never really thought about. You can find this information in bamboo-agent-home/bamboo-agent.cfg.xml. The server uses this information to decide if an agent is new or not.

So far I have not had any issues reusing this UUID identifier... however someone else may feel differently about this approach.  Bamboo will however reuse the name that appears in the interface for that UUID/agent relationship and change it by appending an incremented number on the end - e.g. `agent-name (n)` - for each iteration of that UUID detected.

I included steps for adding a the docker executable to your instance, these steps are pretty much copied verbatim from here https://confluence.atlassian.com/display/BAMBOO/Viewing+a+Bamboo+agent%27s+details

To define a new agent-specific executable capability:

* Click the 
 icon in the Bamboo header and choose Overview.
* Click Agents in the left panel to display the 'Agents' page, which lists all local and remote agents that currently exist in your Bamboo system.
* Click the name of the desired agent. The agent's page will be displayed.
* Click one of the Capabilities tab for that agent.
* In the 'Agent-Specific Capabilities' section of the Capabilities tab, click Add Capability. The 'Add Capability' page is displayed.
* Choose Capability Type > Executable.
* Select the appropriate executable (Custom) from the Type list.
* In the Executable Label, type a name/label  (Docker) for the executable. Bamboo uses this name in the Executables list whenever a task's executable is configured.
* In the Path field, type the path to the installed executable. This will vary depending on the Type you selected in the previous step. (For *nix `which docker` from the command line will yield this info)
 For Ant and Maven, Bamboo requires the path to be the location of the executable installation folder.

Click Add. This will verify whether the executable and path you have specified are valid.

-- Conversely you can do the same for Bamboo itself.

To define a new local server executable capability:

* Click the 
 icon in the Bamboo header and choose Overview.
* Click Server Capabilities in the left navigation panel.
* Choose Capability Type > Executable in the 'Add Capability' section at the end of the page .
* Select the appropriate executable (Custom) from the Type list.
* In the Executable Label field, type a name/label (Docker)  for the executable, which Bamboo presents in the Executable list whenever a Task's executable is configured.
* In the Path field, type the appropriate path. This will depend on the Type you selected in the previous step. (For *nix `which docker` from the command line will yield this info)
 For Ant and Maven, Bamboo requires the path to be the location of the executable installation folder.
* Click Add.

A definite use of plugin would be awesome. I'm actually doing something very close to what you're doing with veewee, vagrant, puppet, etcd and unison. I however use the entirety of the inline script. We chose to use the script method, because adding a command task to plan for each required step in the docker (or in our case veewee) build process proved unwieldy. Having the steps all inline via the script task was nice and organized to look at.

I'm struggling right now with IP address handling in the environments  and whether or not to use DHCP or static.  I'm definitely interested if your group is using the gitflow process. If yes, how do you spin up environments in relation to your branches and repos? I have an idea of alpha, beta, and staging. Alpha would be for master branches, beta would be for release branches and staging would be for develop.

I'll probably ask that question in another thread, but if you willing to share some thoughts on your workflow I would be interested to hear them.

Good Luck