How to create a custom fanout pipelines?

[yaytul]

How can I create custom pipelines that have the same initial pipeline and then fan out from there?

Let's say I have a single pipe that fans out into 3 different custom pipes.

[Philip Hodder]

Hi Yalim,

Can you expand on your use-case a bit? What conditions would cause the pipelines fan out? Perhaps the parallel steps feature is what you're after?

Thanks,

Phil

[yaytul]

Thanks, I'm aware of the parallel steps, May be I can illustrate here...

(I hope the ascii graphic renders okay)

                 +-B-->
                 | 
                 | 
 >---A-->+-C-->--E-->
                 |
                 |
                +-D-->

The idea is to be able to setup a pipeline in such way that I can execute E.g. A,B or A,C,E or A,D or maybe even more fork out from each endpoint B,C,D, or E.

Hope this helps to visualize it.

[yaytul]

Here is what I figured by reading the documentation. This could be achieved with only with 2 repos. First repo holds source code. Second repo holds all possible paths as different pipeline definitions in different branches. Each branch has a pipeline definition that is a path in the graph. At build time First repo --the source code of the project-- must be copied over to the second repo into the branch that reflects the desired pathway. Obvious downside to this configuration is, you have to duplicate some pipeline configuration between branches. One small change in a pathway will cascade to other branches. E.g. If anything in A changes, {A,D}, {A,C,E}, {A,B}, and anything that stems from A must be changed.

Well, do you have an easier solution? :-D

[Philip Hodder]

(Sorry for the delayed response, I was on annual leave)

Hmmm... Why were you unable to define branch pipelines in the same repository?

e.g. 

pipelines:
  branches:
    path-1-branches*:
      - step:
          name: A
          script:
            - echo A
      - step:
          name: C
          script:
            - echo C
      - step:
          name: E
          script:
            - echo E
    path-2-branches*:
      - step:
          name: A
          script:
            - echo A
      - step:
          name: B
          script:
            - echo B
    path-3-branches*:
      - step:
          name: A
          script:
            - echo A
      - step:
          name: D
          script:
            - echo D

This can be defined on any branch in your repository, Pipelines will match the branch name with the associated pipeline branch configuration.

If you'd then like to remove the duplicate usage of steps (such as with Step A), you can use YAML anchors.

e.g.

definitions: 
  steps:
    - step: &step-A
        name: A
        script:
          - echo A

pipelines:
  branches:
    path-1-branches*:
      - step: *step-A
      - step:
          name: C
          script:
            - echo C
      - step:
          name: E
          script:
            - echo E
    path-2-branches*:
      - step: *step-A
      - step:
          name: B
          script:
            - echo B
    path-3-branches*:
      - step: *step-A
      - step:
          name: D
          script:
            - echo D

Does that look like it would work for you?