Dynamic Pipelines, adding a custom pipeline or user entered variables.

[David Boyer]

Our team recently jumped up to Bitbucket Premium in the hope that we could use Dynamic Pipelines to make it easier to track deployments to our many environments, without having to modify our YAML file each time we add or remove one.

We understand that there are two different executions for the dynamic pipeline module and have the first one working as we expect where we can add a new pipeline named after each environment (based on what we've defined in the repository settings).  However, this all falls apart when the second execution scenario (triggered manually) occurs as we no longer have access to the original full pipelines configuration, that we base the dynamically generates ones off.

Our alternative solution was to use the dropdown feature of pipelines (allowed-values) to list out target environments (which we worked out how to do dynamically based on an API call to the environments endpoint).  This again was working for the first execution scenario, but when triggered manually, we've found no method of gaining access to which item the user picked when triggering the pipeline.  Meaning that we can't dynamically replace the `deployment` value.

TLDR;

We need either:
* A way to access the values entered on a manually triggered pipeline so we can respond to those choices.

* Or a way to access the original pipelines configuration during the manual triggered execution.

[Daniil Penkin]

Hello @David Boyer

I'm Daniil from Pipelines dev team.

If I got it right, you're looking to reuse some pipeline definition when generating a pipeline for another selector. In this case, as you described, we do send full static configuration to the Dynamic Pipelines provider when we request all available definitions – this is to allow a user to select the pipeline they want to trigger. But on the second request when the user selected a definition that was added dynamically, we won't send any definition from the static configuration because none of them matched the pattern of the selected pipeline.

There are at least two ways to deal with this:

• you can use top-level labels property in your static config file to define the entities you want to reuse, then use YAML anchors and aliases to apply them in the static file, and read them in your Dynamic Pipelines provider implementation. labels can be considered a generic storage with a size limit of 10KB when parsed (that is, all aliases resolved, if any).
  In this case on the second request the provider would still not get any pipeline definitions cause none matched the selected one, but it will get labels as is.
• another approach is applicable to scenarios where a pipeline definition is to be reused only between different dynamically generated pipelines, but not the static ones: you can store pipeline definitions in separate YAML or JSON file(s) in your repository, and make the Dynamic Pipelines provider fetch them when necessary using Bitbucket API. Such external definitions obviously won't be available in the static YML configuration, but the Dynamic Pipelines provider has no limitations as to what it can do to generate a pipeline.

As to making the Dynamic Pipelines provider aware of the custom variable values entered by the user, this is a feature request we are aware of. I was going to say please vote for BCLOUD-23395 and subscribe to it so that you get notified on any updates to this topic, but I can see you've already done that 👍

Hope this helps. Let me know if you have any questions.

Cheers,
Daniil