Amazing use of Dynamic Pipelines feature

[Shawn Castrianni]

I wanted to share my usage of the Dynamic Pipelines feature of Bitbucket.  I think it is very cool and opens up a whole new world of features for pipelines.

Bitbucket added the ability to write a Dynamic Pipeline plugin which can affect the final pipeline yaml file seen by pipelines before a pipeline is even started.  I believe I have taken huge advantage of that by adding ytt.  ytt is a very powerful yaml template engine.  It essentially provides a complete programming language inside yaml: for loops, if conditionals, variables, function definitions, etc.  It also has the concept of overlays which allows another yaml file to be merged into the main yaml file.

My Dynamic Pipeline plugin reads the bitbucket-pipelines.yml file from the repo, processes it with ytt, and then returns the result to Bitbucket Pipelines.  Therefore, what Bitbucket Pipelines sees is standard yaml, but the bitbucket-pipelines.yml file inside the repo can have all kinds of expressions that is not possible with standard BB.

A very simple example of what you can do is...  Imagine if you wanted a step repeated multiple times such that each repeated step had a different name and the list to be repeated was provided from another file in your repo.

The bitbucket-pipelines.yml file could look like:

#@ load("@ytt:data", "data")

#@yaml/text-templated-strings
pipelines:
  default:
    #@ for env in data.values.environments.split(","):
    - step:
        name: '(@= env @)'
        script:
          - echo "Your build and test goes here..."
    #@ end

All of ytt's language constructs exist inside yaml comments.  When ytt process this file with a supplied data file that gives the value of environments as a comma-delimited string, ytt outputs the final file as:

pipelines:
  default:
  - step:
      name: "dev"
      script:
        - echo "Your build and test goes here..."
  - step:
      name: "stage"
      script:
        - echo "Your build and test goes here..."
  - step:
      name: "prod"
      script:
        - echo "Your build and test goes here..."

By using this, my company maintains a single master pipeline template that all repos use and each repo can tweak that template by supplying their own data values and overrides.

I have requested that Atlassian add builtin support for ytt so that a custom dynamic pipeline plugin is not required.

[plussier]

Wow! This is pretty cool!

Can you explain more in-depth how this works? For example, how is your dynamic pipeline code (which I assume is Typescript?) integrating the use of ytt?

I'm just beginning to explore dynamic pipelines, and I'm not only new to this, but also to TypeScript as well, so pretty steep learning curve for me on multiple fronts.

Thanks!

--

Paul

[Shawn Castrianni]

That was the hard part.  ytt is distributed ONLY as a binary executable.  No traditional npm package is available to be used in a nodejs plugin.  There is a npm-ytt-cli package that seems like it would work, but does not.  I suspect it is because of the forge security restrictions on what plugins are allowed to do.  I can only imagine the issues if BB allowed a plugin to run whatever executable they wanted.

Therefore, I had to write my own REST API that wrapped the ytt binary so that my BB dynamic pipeline plugin just had to make a simple REST API query to handle the ytt processing.