Automating setup of a build plan clone that is targeted to a new release branch

[Ed Segall]

I'd appreciate some assistance with a design decision I have to make for automating a manual Bamboo operation.

When cloning a build plan in Bamboo, several fields must be manually configured, in multiple stages. These fields include project, plan name, plan key, and repository info (where the repository info consists of multiple fields). In our release branch workflow, these values can all be calculated from the original plan configuration plus the new branch identifier. For a variety of reasons, we would like to automate this process.

I looked at the REST and Remote APIs, but I don't see a way to copy or create plans using either one. I've identified several possible alternative approaches that seem to be feasible, but I haven't been able to find enough information to be confident that I can make a good choice about which of these would actually work, nor that I have enough information to make it (or them) work.

1) Using Curl or another command-line client, mimic the form submissions made by the Bamboo web interface when a user manually clones a plan and sets up its unconfigured parameters.

2) Write a Bamboo plugin using the Java API to create a new plan, copy values from the original plan where applicable, and initialize the other fields to the desired values.

3) Use direct database operations from an external script to implement the same operations as in #2.

Option 1 seems quickest and simplest, and also most susceptible to having to be revised in the future. That may be OK, especially if it's a lot faster than the others. It also seems very likely to work, since it would do the same things that the GUI does. Assuming of course that there are no subtle gotchas hiding somewhere.

Option 2 seems to be the long-term best approach, but possibly the most heavyweight. (Probably very quick for someone who's developed a Bamboo plugin before, but IIRC getting started on Jira plugins was very time-consuming so I'm not optimistic that I can do Option 2 very quickly.) The only info on this that I've been able to find is at http://docs.atlassian.com/atlassian-bamboo/latest/. The descriptions there are very terse, so it's not clear to me that the API calls behave the way I am guessing they do. PlanManager's createPlan() seems to be a good place to start, but it would be nice to have more information about how it works and how to use it.

Option 3 seems straightforward, and less complicated than Option 2. My main concern is that there may be things done when a plan is created that are not obvious by looking at the database. Thus if I go behind the server's back to update the database, I could possibly confuse it. Again, this method is susceptible to needing revision in the future, although that seems less likely (or at least more straightforward to manage) than Option 1.

Any insight you can provide would be appreciated.

Thank you!

--Ed

[Ed Segall · Accepted solution]

Since I received no response regarding any of the proposed options, I elected the most expedient path and implemented Option 1. Direct database access was necessary to extract the Git repository configuration. Other than that it was all done via curl.

--Ed

[Ed Segall]

Adam,

I would recommend against using direct database access to modify the database. In my implementation, I used direct access where necessary to query the database, but I made all modifications via the web UI. I did this to avoid bypassing any concurrency control or consistency maintenance that may be managed by Bamboo. (Direct query still risks reading inconsistent data, but at least this approach avoids writing inconsistent data or putting the database in a state that's inconsistent with the app's state.)

A few minor updates were needed when we upgraded Bamboo from 3.1.1 to 3.4.4 (including one that only came up occasionally so we only tracked down its cause recently) but the changes needed weren't hard to deal with.

Also, have you looked at the support provided recently for branches? It came out after my implementation was in production, so I haven't spent time getting familiar with it, but it appears to address similar issues to the ones we had.

--Ed

[Ed Segall]

Adam,

I would recommend against using direct database access to modify the database. In my implementation, I used direct access where necessary to query the database, but I made all modifications via the web UI. I did this to avoid bypassing any concurrency control or consistency maintenance that may be managed by Bamboo. (Direct query still risks reading inconsistent data, but at least this approach avoids writing inconsistent data or putting the database in a state that's inconsistent with the app's state.)

A few minor updates were needed when we upgraded Bamboo from 3.1.1 to 3.4.4 (including one that only came up occasionally so we only tracked down its cause recently) but the changes needed weren't hard to deal with.

Also, have you looked at the support provided recently for branches? It came out after my implementation was in production, so I haven't spent time getting familiar with it, but it appears to address similar issues to the ones we had.

--Ed

[Adam Downer]

hi Ed, thanks for the advice. I have been looking for a method that does not require direct DB manipulation as it is nasty. When you say "I made all modifications via the web UI." do you mean manually or as part of a script. if this was scripted I would be interested in hearing your method. My query is slightly different to the original above so branch support doesn't really cover my use case. see here

[Ed Segall]

Adam,

It's done as part of a script.

Using Chrome's developer tools, I reverse-engineered the http form parameters that are specified when modifying the repository configuration. I then set up a script function that uses curl to do the same thing.

In Bamboo 3.4.4 (but not in 3.1.1; i.e. this is one of the changes that was needed after upgrading), one of the required parameters is called repositoryId. The script gets it via a database query for the plan's VCS_LOCATION_ID field.

I see from your link that you're working with Bamboo 4.2.1. Hopefully it's not too different in this regard.

--Ed

[Sai N]

Ed

After all these years.. I am trying to do the same. We are cloning  a build plan using a script. But for editing the just created clone i am going to the Bamboo UI to get it's environment ID. 

How did you achieve this in your script? you said you used a database query to do something similar.

Please help.

TIA

[Adam Downer]

its just over a year later and I am also implementing option one. The latest version of the rest API and CLI tool still do not (as far as documentation states) not allow access to update the repository location so direct database access is the way I am going as well.

My solution is this:

use bamboo cli
clone a template project with new name.
update and/or add required tasks
use direct DB
get PLAN_ID
get VCS_LOCATION_ID where PLAN_ID equals above
update VCS_LOCATION set XML_DEFINITION_DATA with new repo location where PLAN_ID equals above

[Adam Downer]

The downside of this approach comes with updating the system etc. The schema structure will no doubt change and break the script. But this will be tested somewhere before updating, of course. I would love to extend the rest API or the bamboo cli tool to rule out the direct DB connection, but my own recent query on this has not turned up any newer information and I do not have the skills to write my own plugin.

[tkhduracell]

I'm using #1 to setup tests in our automated environment.

11-jul-2012 18:09:41	-- Creating test: 1_platform_SR ==> AT6WI1PLATFORMSRJUNT 
11-jul-2012 18:09:42	        Put job 1_platform_SR[AT6WI1PLATFORMSRJUNT] onto plan VER12TEST-MAIN:Test Stage
11-jul-2012 18:09:45	        Put task Script:install-server.bat [OK id: 1]
11-jul-2012 18:09:48	        Put task TFS-Task GET $/sadasdadsad/test/Platform [OK id: 2]
11-jul-2012 18:09:51	        Put task Script:[JUnit] 1_platform_SR [OK id: 3]
11-jul-2012 18:09:54	        Put task JUnitParser:**/results/**/*.xml,**/results/*.xml [OK id: 4]
11-jul-2012 18:09:57	        Put task Script:test-cleanup.bat [OK id: 5]
11-jul-2012 18:09:59	        Defined artifact Result to job AT6WI1PLATFORMSRJUNT
11-jul-2012 18:10:01	        Moved final bar, 4 -> 3
11-jul-2012 18:10:01	        Put requirement testAgent=true to job AT6WI1PLATFORMSRJUNT
11-jul-2012 18:10:02	        Artifact Suite.exe resolved to id 8093697
11-jul-2012 18:10:02	        Put artifact dependency Suite.exe [8093697]
11-jul-2012 18:10:03	        Artifact scripts resolved to id 3702790
11-jul-2012 18:10:03	        Put artifact dependency scripts [3702790]
11-jul-2012 18:10:07	        Setting clean_workdirectory = true

I've build a Java program consuming the action rescources Bamboo web-ui uses, and use it to set upp a job per testcase to parallaize testing across machines.

[tkhduracell]

I've been doing this to set up jobs dynamically for commited tests. I find this very hard at the moment, but I've succeeded kind of, ugly but it works:

public class BambooClient{
...
public boolean putJob(Plan p, Stage s, Job job) {
        Client client = Client.create();
        WebResource res = client.resource(server+"chain/admin/createJob.action");
        
        MultivaluedMap<String, String> map = getLoginMap();
        
        map.add("existingStage", s.key());
        map.add("buildKey", p.key());
        map.add("buildName", job.name());
        map.add("buildDescription", job.toString());
        map.add("subBuildKey", job.key());
        map.add("confirm", "true");
        map.add("bamboo.successReturnMode", "json");
        
        String ans = res.queryParams(map).get(String.class);
        
        info("Put job "+job.name()+"["+job.key()+"] onto plan "+p.key()+":"+s.key());
        checkError(ans);
        return true;
        
    }
}

I will have to do this with dynamic plans later on, but this works for static plans for now.

And I hope this features will be avalible in later releases of Bamboo

[Ed Segall]

Bob,

Yes, if I were to write a plugin, one option would be to have it expose a Rest API. That's what I did with a Jira plugin that I wrote. To do this, though, I'd need to have more information about the Bamboo Java API, I as mentioned. Either that, or guess and hope for the best.

Thanks for letting me know about the Bamboo CLI. It looks like a useful tool.

--Ed

[Bob Swift OSS (Bob Swift Atlassian Apps)]

One of the goals of Bamboo Command Line Interface is to provide this kind of capability. However the Bamboo provided rest apis are very limited currently. It is hoped those will be expanded in the future to cover this kind of automation need. In the meantime, Atlassian usually suggests to have 3rd parties extend the rest api to cover additional support using their guidelines. So, an option for you would be to write a Bamboo plugin to expose additional rest apis to accomplish your goals. Then potentially the Bamboo CLI can be extended to use these or you could use them from curl. The advantage over option 1 is to avoid compatibility problems with web interface changes.