It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Best way to reboot agents in Bamboo?

Does anyone have some best practices for a scheme that automatically reboots remote agents? I'm thinking of adding a cron job on the main Bamboo server that loops over all agents, disables them, waits for the current build to finish (if there is one), reboots, waits for the machine to come back, and enables it.

Is there a better way that escapes me?

4 answers

1 accepted

1 vote
Answer accepted

We manage hundreds of remote agents for Atlassian's internal build system. Most of then Linux. We hardly ever need to reboot them. Though that's probably because we try to have builds be 'well-behaved' in the sense that they tear down any transient resources they might create while they run.

The agent processes themselves we manage with daemontools so we could easily loop over all agents and do:

$ svc -d /service/bamboo-agent

followed by

$ shutdown -r now

Although if there are no builds running just calling 'shutdown' is sufficient since daemontools will ensure the 'service' comes back up on restart. You would just have to make sure daemontools is always running in your /etc/rc.local configuration

We also use Puppet to manage the state of all those agents. So adding the cronjob definition uniformly across all agents would require 1) writing the Puppet code to do it 2) updating the puppetmaster

Since Puppet ensures that daemontools is always running then calling 'shutdown' with Puppet present is also sufficient.

Other ways to loop over your agents including using cluster-SSH and Fabric. We use both for adhoc orchestration of maintenance tasks.

Dealing with the agents directly is fairly straightforward, the tricky part even for us is negotiating with Bamboo for a 'window of opportunity' to shut the agents down. Przemek can correct me but since Bamboo 3.3 there should be a way to PAUSE the server. I think there might even be a REST endpoint to do that.

What command(s) do you use for your daemontools bamboo-agent run script?

https://marketplace.atlassian.com/plugins/com.edwardawebb.bamboo-agent-apis

Open source and free until Atlassian provides a more elegant solution. Uses security tokens instead of standard credentials to allow scripted interactions without exposing sensitive accounts.

Even has handy bash-like output

0 votes

An alternative approach to doing this (although requiring a bit of work if you have a lot of agents, you must have a bored intern for that if you have a 100 agents license):

- create a plan with a job that executes a script task (a file based one, it will be easier to modify it later) that disables the agent remotely and does the reboot in background after ~2-3 seconds. A disabled agent will finish its current job, but will not pick up a new one. If you don't fork off the reboot, the build will be failed.

- Clone this job within your plan to have as many jobs as you have agents

- assign a unique capability to each of your agents

- add a requirement to a unique agent to each Job

It should be simpler to set up, but if you often set up new agents or have lots of agents to begin with, I'd rather go with the approach you've described. Also, the remote disabling of agents may be a security risk if not secured properly (just like the remote reboot in your approach).

Since I don't have a bored intern at my disposal, I'm going with the cron job. One thing I needed to do that wasn't in my initial question is that the rebooting process writes a small temp file on the machine with the current state of the agent (enabled or disabled). That way, when the machine starts up again the cron job runs and looks for the temp file. This way it won't inadvertently enable a machine that was already in a disabled state.

@przemek-

" that disables the agent remotely "

Have any advice on that step? I dont see any APIs or agent commands to disable agents.

There's no API, you have to POST to the action that disables the agents.

until now!

https://marketplace.atlassian.com/plugins/com.edwardawebb.bamboo-agent-apis

Open source and free until Atlassian provides a more elegant solution. Uses security tokens instead of standard credentials to allow scripted interactions without exposing sensitive accounts.

Even has handy bash-like output

To automatically manage agents I would suggest a free and open plugin I recently released that includes both json and bash-like output for easy automation.

Use of security tokens keeps sensitive credentials out of scripts.

https://eddiewebb.atlassian.net/wiki/display/AAFB/Remote+Agent+Management+with+monit

This page was deleted.

See https://eddiewebb.atlassian.net/wiki/display/AAFB/User+Guide for various examples including calls from monit and cron

Suggest an answer

Log in or Sign up to answer
Community showcase
Published in Bamboo

Unable to add or edit Bitbucket Cloud repository in Bamboo

On 31 May, a GDPR-related change went live in the Bitbucket Cloud API that resulted in users not being able to create or edit Bitbucket Cloud Linked repositories in Bamboo. This API update removed t...

1,154 views 2 7
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you