Can we have multiple Bamboo YAML specs and services in one repository?

[Lars Kristian]

Currently we are using the monorepo strategy where we have the source code for multiple services in one repository.

When looking into YAML build plans, it seems like it's limited to one spec pr repository, ref. https://confluence.atlassian.com/bamboo/tutorial-bamboo-specs-yaml-stored-in-bitbucket-server-941616819.html

It's important to save your Bamboo Specs YAML definition in the ${repo-home}/bamboo-specs/bamboo.yml or bamboo.yaml  file under the repository root.

Ideally we would want to gather the bamboo.yaml and source code for the different services into different folders and then trigger builds only if there are any changes in that specific service's folder.

Is this doable? I couldn't seem to find any documentation/questions related to monorepo and YAML builds.

[Victor Debone · Accepted solution]

Hello @Lars Kristian, welcome to the community :)

The feature you are looking for will be released in the next version of Bamboo.

On Bamboo 6.9, you will be able to declare multiple plans and deployments projects, using one or more files. You can try and experiment with it now, with the EAP version of 6.9 (link).

If you want to share more details, give some feedback on your current usage (or future) of Bamboo Specs with YAML, feel free to send a message at vdebone@atlassian.com :)

Victor

[Lars Kristian]

Thanks for the quick reply @Victor Debone!

What I imagine is something like this

• Jobs
  • JobA
    • docs/
    • src/
    • scripts/
    • readme.md
    • bamboo.yaml
  • ...
• Services
  • ServiceA
    • docs/
    • src/
    • scripts/
    • readme.md
    • bamboo.yaml
  • ServiceB
    • docs/
    • src/
    • scripts/
    • readme.md
    • bamboo.yaml
  • ...
• ...
• readme.md

How would the build plans be visualized/organized in Bamboo 6.9? Given that the monorepo above is branched, would you still have one build plan based on the YAML project.plan.key, where the steps then might differ from branch to branch if changes are made to bamboo.yaml? Or would you have copies of the build plan per branch which might lead to overloading the build dashboard? (I'm hoping the former since one usually creates a branch to create or make changes on one service, and there would be no point in creating duplicates of all the build plans. The build dashboard might not even be navigatable.)

Any information on when the 6.9 version is scheduled for GA? I don't think there's a possibility of getting the EAP installed for now, unfortunately. (The "link" above is not a link at the moment btw.)

[Victor Debone]

Great! 

I will answer them in reversed order.

To make it clear, the EAP version is not for production :) My suggestion of EAP was only if you wanted to check the new YAML first-hand! And for the release date, we don't keep public facing dates for releases.

On the YAML part, plan branches differing will not be part of this release. Bamboo Specs on YAML (and will for 6.9) only accepts the master branch `bamboo.yml`. Definitely this is under our radar, it is high priority for us, and you can follow its progress on this suggestion.

For the organization of the repo, this is what will be possible:

Repository
├── your-service-a
│   ├── files
│   └── ...
├── your-service-b
│   ├── files
│   └── ...
└── bamboo-specs
    ├── bamboo.yml
    ├── your-service-a.yml
    └── your-service-b.yml

Would that work for you? (Apart from the lack of differing branches configurations)

Victor 

[Lars Kristian]

Thanks,

For the repository structure, would that mean the high-level bamboo.yml have some kind of mapping saying which path belongs to the different Bamboo yml-files? So if there are changes in the path, trigger pipeline X. (For instance there is no point in running builds for documentation changes, only application code.)

your-service-a/service-a-sourcecode => your-service-a.yml

your-service-b/sourceCode => your-service-b.yml

You are saying that the YAML build plan will only accept the master branch's bamboo.yml. Does that mean we can't even duplicate the plan configuration to a new one where the branch is explicitly set to the feature branch if we need to work on the YAML build plan(s)? We are only allowed to do that directly in the master branch?

[Victor Debone]

If I understood your first question correctly, yes. You will have a single bamboo.yml file that will include all the others files, for organisation purposes, and they are required to be inside the `bamboo-specs` folder. This is how it might look:

---
!include your-service-a.yml

---
!include your-service-b.yml

...

And from each of these files you will be defining either a plan or a deployment. 

On the second question, yes. Bamboo 6.9 will still only support the master's branch `bamboo.yml`, and again, this is a high priority suggestion for us and you can follow it's progress under BAM-19620

Did that help? :)

Victor 

[Lars Kristian]

Thanks, that clear things up.

The only thing I'm not seeing is how the relationship is set up between the yaml file and the source code location in terms of triggering a build.

From spec-yaml it should be simple to have relative paths pointing from "your-service-a.yml" to for instance the directory where "your-service-a" resides for the scripts, test results etc.

But where do you define the trigger path? In other CI tools they call this "path filter" or similar. It usually means: "ok, if something is changed/added/modified in path x/y/z then the build definition that should be run is z.yml". I'm assuming that all the plans aren't triggered if just one file in a certain service directory is changed?

[Victor Debone]

Glad that I could help!

For the "Path filter" feature, I have just created a feature request BAM-20422. We currently do not support this granularity of change detection natively.

It's not 100% clear what is your urgency on that, but I also want to leave it stated that we have plugin points at change detection and writing a plugin to cover this gap would be an alternative.

Did that help?

[Lars Kristian]

Thanks for all your help,

To summarize, our use case isn't really supported at this time, but will likely be with 6.9 + BAM-20422. So for this question itself, I'll close it as answered.

As to the urgency, we'll look into workarounds and/or other tooling. If you can link to documentation regarding the plugin point you mentioned, that might help as well, but I'm not sure we have the time to go that route.

[Matthew Lieder]

@Victor Debone did this get dropped from 6.9? We're on 6.9 now and I'm not seeing it mentioned in documentation or working when I try.

[Victor Debone]

@Matthew Lieder which feature are you specifically trying out that is not working?

[Matthew Lieder]

@Victor Debone my initial reading of this thread seemed to indicate that putting "plan-1.yml", "plan-2.yml", etc. inside a bamboo-specs folder would be all that's needed in v6.9 to having multiple plans. After further digging I found hidden inside the "Understanding YAML in Specs" section at the bottom of the docs (https://docs.atlassian.com/bamboo-specs-docs/6.9.0/specs.html?yaml#understanding-yaml-in-specs) details on how to import specs from the root spec (bamboo.yml) and also, more importantly to me, how to combine multiple specs into one file (separating them by ---). So, while the most straightforward scenario (multiple simple spec files) doesn't work, there are workable alternatives. After re-reading this thread I realized you were in fact referring to the !import method yourself too. Sorry about misreading that!

[Victor Debone]

Glad that you found the solution! 

Indeed the documentation is misleading, specially in this regard, and feedback was already taken into consideration for updating it.