how to use bamboo specs include (YAML)

[Robert Kulyassa]

Hi all,

I've started to play with bamboo yaml specs. It's very promising, but I found some issues. One of them the `

I try to include multiple jobs from a file. I have jobs/all_jobs_in_one.yaml:

job1:
   ....
   ....
job2:
   ....
   ....

and I try to include it to my bamboo.yaml like this:

---
version: 2

plan:
   ...
   ...

stages:
   ...
   ...


!include 'jobs/all_jobs_in_one.yaml'

but I got the following error:

01-Jan-1970 01:00:00 Bamboo YAML import failed: Invalid format of the YAML file: while scanning a simple key
01-Jan-1970 01:00:00 in 'reader', line 37, column 1:
01-Jan-1970 01:00:00 !include 'jobs/all_jobs_in_one.yaml'
01-Jan-1970 01:00:00 ^
01-Jan-1970 01:00:00 could not find expected ':'
01-Jan-1970 01:00:00 in 'reader', line 39, column 1:
01-Jan-1970 01:00:00
01-Jan-1970 01:00:00 ^
01-Jan-1970 01:00:00

(39 is the very last line)

If I don't use the include, but copy&paste the content of the all_jobs_in_one.yaml into the bamboo.yaml  it works perfectly.

In the example of the include they use this format:

job1: !include job1.yaml
job2: !include job2.yaml

which works fine for me too, but what if I want to include more?!

The doc says about include:  "This tag is flexible, and it can be used anywhere in the file, as long as it follows the YAML format."

Is it a bug or I use it on a wrong way?

[Victor Debone · Accepted solution]

Hello @Robert Kulyassa

First, it's great to hear that you are using Bamboo Specs YAML :)

For your use-case, the snippet you have found on the docs with the example of include is the current way to go. This is a current limitation of the `!include` tag.

The documentation indeed mentions that it should follow the YAML format and may be using a loose language, while in fact we are very strict about the "should follow YAML format".

As the documentation, the `!include` in Bamboo Specs is an YAML tag. By definition, and according to the specification of YAML, tags are treated as scalars. In YAML, scalars are strings or numbers, for example. The include of other files happens after the YAML document is parsed. This is the cause of the limitation you are hitting.

To get a practical view, if you are writing a YAML document:

---
version: 2
plan: ...
stages: ...

You are writing a YAML Map, which is a key-value map.

If you are thinking about a key-value map, it would make sense that this would be an invalid YAML document:

---
version: 2
plan:
  ...

some-string-not-related-just-here

Causing the parser to complain about:

could not find expected ':'

Because we didn't close the key "some-string-not-related-just-here" with ":" and didn't define its value.

Looking back at the use of "!include":

---
version: 2

plan:
   ...
stages:
   ...

!include 'jobs/all_jobs_in_one.yaml'   #<-- The key '!include...' is left without ':'

You can understand why the parser was also breaking with the use of "!include" tag on the root level of the map.

That also shows why:

job1: !include job1.yaml
job2: !include job2.yaml

Is a valid use. The key-values are clearly defined.

For even more details, you can dive in into the YAML specs here

---

Also, I will highlight that this answer is not a closing-the-topic-cannot-be-done answer, but more a why it works the way it works currently for Bamboo version 6.9.0 and Bamboo Specs YAML 2.0, there are ways to improve the usability of single file imports.

We are aware of these limitations and we always welcome feedback, like this one you have provided, for us to understand more how this limitation impacts our users. If you have more details on how this impacts you, and how you are using other pieces of Bamboo Specs YAML, feel free to drop more comments here or just send an email at vdebone@atlassian.com :)

[Robert Kulyassa]

Hi @Victor Debone

Thanks for the extensive description. Include is a must for me to reduce redundancy... one of the reason why I try to introduce in our build system  the YAML specs instead  of the web based configuration.

You asked for other example of usage, so I give you one. :) I have lot's of similar jobs (not only 3 as in this example):

job1:
  requirements:
         - !include same.for.all.jobs.in.the.plan.yaml
   tasks:
         - !include sametask1.yaml
         - !include sametask2.yaml
         - !include sametask3.yaml
         - !include sametask4.yaml

job2:
  requirements:
         - !include same.for.all.jobs.in.the.plan.yaml
   tasks:
         - !include sametask1.yaml
         - !include sametask2.yaml
         - !include sametask3.yaml
         - !include sametask4.yaml
         - !include different.task.yaml

job3:
  requirements:
         - !include same.for.all.jobs.in.the.plan.yaml
   tasks:
         - !include sametask1.yaml
         - !include sametask2.yaml
         - !include sametask3.yaml
         - !include sametask4.yaml
         - !include another.different.task.yaml

They are 90% the same, but because of the small differences and because of the  behavior of the include (what you just described) I can not simplify it more.
I can put the 4 same tasks to 1 yaml and I am able to include it, but then I can not extend that list with a new task , so the

   tasks:
         - !include multiple.tasks.in.one.file.yaml
         - !include different.task.yaml

doesn't work.

Anyway, the yaml specs rocks! I really missed this feature before. I see lot's of holes, but it already helpful. Don't  give up, extend it more!
First focus on the missing branching, not on the include. ;)

Thanks,
--
Robi

[Victor Debone]

Hey @Robert Kulyassa, thanks for the feedback!

I have created a suggestion for this use-case and I will keep track of it. Sequences in YAML are not mergeable, but it would make sense to me to allow nested arrays of tasks to work as it would be a single array. This would allow to constructs like the one you've suggested to work:

tasks:
  - !include multiple.tasks.in.one.file.yaml
  - !include different.task.yaml

Let's see how this will attract attention at BAM-20497

[Jurn van Mieghem]

Could you also give an example of what sametask1.yaml would look like?