Best way to reboot agents in Bamboo?

Does anyone have some best practices for a scheme that automatically reboots remote agents? I'm thinking of adding a cron job on the main Bamboo server that loops over all agents, disables them, waits for the current build to finish (if there is one), reboots, waits for the machine to come back, and enables it.

Is there a better way that escapes me?

4 answers

1 accepted

Accepted Answer
1 vote

We manage hundreds of remote agents for Atlassian's internal build system. Most of then Linux. We hardly ever need to reboot them. Though that's probably because we try to have builds be 'well-behaved' in the sense that they tear down any transient resources they might create while they run.

The agent processes themselves we manage with daemontools so we could easily loop over all agents and do:

$ svc -d /service/bamboo-agent

followed by

$ shutdown -r now

Although if there are no builds running just calling 'shutdown' is sufficient since daemontools will ensure the 'service' comes back up on restart. You would just have to make sure daemontools is always running in your /etc/rc.local configuration

We also use Puppet to manage the state of all those agents. So adding the cronjob definition uniformly across all agents would require 1) writing the Puppet code to do it 2) updating the puppetmaster

Since Puppet ensures that daemontools is always running then calling 'shutdown' with Puppet present is also sufficient.

Other ways to loop over your agents including using cluster-SSH and Fabric. We use both for adhoc orchestration of maintenance tasks.

Dealing with the agents directly is fairly straightforward, the tricky part even for us is negotiating with Bamboo for a 'window of opportunity' to shut the agents down. Przemek can correct me but since Bamboo 3.3 there should be a way to PAUSE the server. I think there might even be a REST endpoint to do that.

What command(s) do you use for your daemontools bamboo-agent run script?

https://marketplace.atlassian.com/plugins/com.edwardawebb.bamboo-agent-apis

Open source and free until Atlassian provides a more elegant solution. Uses security tokens instead of standard credentials to allow scripted interactions without exposing sensitive accounts.

Even has handy bash-like output

0 votes

An alternative approach to doing this (although requiring a bit of work if you have a lot of agents, you must have a bored intern for that if you have a 100 agents license):

- create a plan with a job that executes a script task (a file based one, it will be easier to modify it later) that disables the agent remotely and does the reboot in background after ~2-3 seconds. A disabled agent will finish its current job, but will not pick up a new one. If you don't fork off the reboot, the build will be failed.

- Clone this job within your plan to have as many jobs as you have agents

- assign a unique capability to each of your agents

- add a requirement to a unique agent to each Job

It should be simpler to set up, but if you often set up new agents or have lots of agents to begin with, I'd rather go with the approach you've described. Also, the remote disabling of agents may be a security risk if not secured properly (just like the remote reboot in your approach).

Since I don't have a bored intern at my disposal, I'm going with the cron job. One thing I needed to do that wasn't in my initial question is that the rebooting process writes a small temp file on the machine with the current state of the agent (enabled or disabled). That way, when the machine starts up again the cron job runs and looks for the temp file. This way it won't inadvertently enable a machine that was already in a disabled state.

@przemek-

" that disables the agent remotely "

Have any advice on that step? I dont see any APIs or agent commands to disable agents.

There's no API, you have to POST to the action that disables the agents.

until now!

https://marketplace.atlassian.com/plugins/com.edwardawebb.bamboo-agent-apis

Open source and free until Atlassian provides a more elegant solution. Uses security tokens instead of standard credentials to allow scripted interactions without exposing sensitive accounts.

Even has handy bash-like output

To automatically manage agents I would suggest a free and open plugin I recently released that includes both json and bash-like output for easy automation.

Use of security tokens keeps sensitive credentials out of scripts.

https://eddiewebb.atlassian.net/wiki/display/AAFB/Remote+Agent+Management+with+monit

This page was deleted.

See https://eddiewebb.atlassian.net/wiki/display/AAFB/User+Guide for various examples including calls from monit and cron

Suggest an answer

Log in or Sign up to answer
Community showcase
Published Monday in Jira Ops

Jira Ops Early Access Program Update #1: Announcing our next feature and a new integration

Thanks for signing up for Jira Ops! I’m Matt Ryall, leader for the Jira Ops product team at Atlassian. Since this is a brand new product, we’ll be delivering improvements quickly and sharing updates...

500 views 0 9
Read article

Atlassian User Groups

Connect with like-minded Atlassian users at free events near you!

Find a group

Connect with like-minded Atlassian users at free events near you!

Find my local user group

Unfortunately there are no AUG chapters near you at the moment.

Start an AUG

You're one step closer to meeting fellow Atlassian users at your local meet up. Learn more about AUGs

Groups near you