Run cleanup script after any Bamboo build

[Jan Nylund]

We have an issue that sometimes developers are not cleaning up properly, and there might be orphan docker containers kept alive on our Bamboo build agents. (started from the build job).

Is there anything in place that I can use for this?

A simple solution to this would be if Bamboo had a trigger similar to the final-task, that could be configured for example in `bamboo-agent.cfg.xml`.

[Daniel Santos]

 Hi @Jan Nylund,

I think I would try to create a final stage on any build that runs those containers to make sure that they are cleaned at the end of the build.

You can use a final stage for that. Final stages are present as a beta feature in Bamboo since version 5.14. It will become a native feature in Bamboo 5.8+ as you can see from [BAM-9771] Final Stage for plans - Create and track feature requests for Atlassian products.

This is how you can enable the dark feature: 

curl -X PUT <server_url>/rest/admin/latest/darkFeatures/bamboo.final.stages -u <user>:<password> -H "Content-Type: application/json" -d'{"key":"bamboo.final.stages","enabled":true}'

Another option I can think is trying an external solution based on the OS of your remote agents. It could be a service or cron job that would constantly look for those containers running for a longer time and killing them.

I hope that helps.

[Jan Nylund]

Thanks @[deleted]

1. The Final stage solution would mean updating 200+ build plans. At least 50 of these are created with bamboo-specs, which make them readonly. So this wont work. I would love to see this on a global level though. (both a pre and post build plan / deploy plan)

2. Listening to the atlassian log would work in theory, if I'm fast enough to find it before a next job is started. During our rush hours, there's no free time, so in practice this is not doable. A cron job is too slow and would likely collide with the running builds.

[Daniel Santos]

Thank you for explaining. I imagined it wouldn't be an easy change, but that was what I had to share.
I'm curious about something:

• Do you have a script task to run those builds or just a docker task?

Bamboo should not allow a docker task to keep running after the build has ended.

Also, I want to share an idea:

• What if the container itself had a mechanism to kill itself after some time of inactivity?

I'm sharing some thoughts so you can have more options, but I think this should not be happening in the first place.

[Jan Nylund]

We primarily use script tasks since Bamboo tasks tend to cause more issues than they gain. (I would really love to go towards a nice simple yaml like in bitbucket pipelines)

Sometimes Bamboo fires up a testcase using docker-compose, which spawns a network, etc. I wrap this in a shellscript that is using a trap that force cleans up, but we still see cases when this is not handled and this leaves me with a bunch of wild docker containers that Bamboo no longer can see.

Basically, something similar to the checkbox "Clean working directory after each build" that would allow running a custom script taks would solve this problem. :)

[Daniel Santos]

Hi @jan,

I got it, thanks for explaining. Maybe for this specific scenario, you would get some benefits from using docker tasks. It is not really easy to be safe on error handling using OS scripts. They are not designed in a way to make this an easy task. I mean, you can wrap everything up, but that would require some special attention depending on how complex is your script.

Regarding your suggestion, do you want to create a feature request for the suggestion you just gave?

[Jan Nylund]

Yes please do. Please request it for both global and plan level. In it's simplest way it could be used to simply echo timing information for log aggregation. (which means it would be nice to have a before start as well) :)

[Daniel Santos]

Hey, I was writing the feature request about this and realize that this is not that simple. I will need your help to better write this request.

Please confirm if my understanding is correct. Wath I understood from your opinion is that we should have a mechanism where a given task could be configured to run for every building plan, right?

Some things to consider:

• What could be the agent running this task?
• If the agent is not the same as the one(s) for the given plan, are you still thinking this would be helpful?
• What would be the OS running this task?
• Considering that some plans would use Windows and others Linux, is this still valid to have a task that will run on a different agent globally for Bamboo?