Add new pipeline using dynamic pipelines

[Alexander James]

Hello,

I am trying to add extra custom pipelines dynamically using dynamic pipelines. This is meant to allow selection of runners for a particular pipeline.

The Forge app:

1. Reads the repository variables

2. For each variable starting with "RUNNER_", it adds a new custom pipeline with the runner name, so in the pipeline selector dropdown list they appear as "custom: runner_name_x" 

While this seems to generate a valid request response, and I am able to select the generated pipeline from the selector the pipeline does not run. It gives me the error of "No matching pipeline definition in the Pipelines configuration."

It looks like the Forge app returns the correct definition of the pipelines to the selector, it does not seem to reflect when it's actually run. I have followed the examples to add steps to an EXISTING pipeline and these work perfectly.

Also, if I add a hardcoded pipeline name ("runner_name_x" in the example above) into bitbucket-pipelines.yml file, then I can successfully execute the pipeline.

(If there is an easier way to select a particular runner for a pipeline at run-time, then please let me know. Doesn’t seem like I can read selector variables upon pipeline submission to allow me to specify the “runs-on” parameter.)

Can anyone advise?

[Garrett_LoVerde · Accepted solution]

See https://community.atlassian.com/t5/Forge-for-Bitbucket-Cloud/User-input-for-dynamic-pipelines-amp-selecting-a-particular-self/td-p/2725138

I still don't have a solution I like.

[Alexander James]

Yes, I saw that one. It seems ridiculous that you cannot easily select a runner at run-time. I considered a few ideas:

1) Select runner from dropdown list and use that as the input tag to "runs-on" - the "runs-on" parameter does not accept variables

2) Query the pipeline to retrieve the runners and create duplicate pipelines with the runner I want. BB doesn't have an API to query runners(!). 

3) Create a variable for each runner to maintain a list and create pipelines as per option 2. This works but the pipelines don't run... 

What solution do you have at this stage? For me, creating a usable pipeline dynamically would be a good start...

[Daniil Penkin]

Hello @Alexander James,

Thanks for reaching out!

When you add custom pipelines, there should be two requests to the Dynamic Pipelines provider involved:

1. The first one – with trigger: definitions – is to get all available pipelines to offer them to the user in the dialog you mentioned. It sounds like your app returns this response, and your dynamic custom pipelines do appear in the dialog ✅
2. Once user selects a pipeline from the list, we actually fire another request to the provider to get definition of that selected pipeline. This time the request will specify trigger: manual and the selector specifying user selection, and we expect the provider to return a single matching pipeline definition which we will use to kick off that pipeline.

I understand this is somewhat confusing, the reason for making two requests is to cover different trigger pipeline scenarios. For this particular scenario we're effectively using only pipeline names and variable definitions from the first request (to show the dialog), while use the actual pipeline definition from the second request.

Does this make sense?

I believe the error you see is because on the second request we didn't get the definition matching the selector. Basically, second response should contain pipelines.custom.<selected_pipeline> subtree so that we can match it.

Let me know if I got it right, and this was the issue.

Cheers,
Daniil

[Alexander James]

Thank you Daniil, this is the exactly what the issue was.

My code was only adding a pipeline definition on if (pipelines.custom) condition. When I checked the trigger: manual request, pipelines.custom didn't exist, so it was never added and returned an empty response (or rather, unchanged).

I don't know if this is the right way to do this but it seems to work OK.

[Daniil Penkin]

Hey @Alexander James, I'm glad we identified the problem.

What do you mean by if (pipelines.custom) condition? Is it actually pipelines_configuration.pipelines.custom? In this case, this is not quite right.

In pipelines_configuration we send the matched configuration, if any, from the static YML file. Dynamic Pipelines provider can alter this configuration in any way. So presence of pipelines_configuration.pipelines.custom means that we matched that definition to the user input, in case the pipeline run is being request, OR we're asking the provider to return all possible pipeline definitions for the user to select from.

This is why the first thing to check in the request is trigger property. In case it equals definitions, we want the provider to return all pipelines a user can run – essentially, we're about to show this dialog and need to fetch the items for this dropdown:

We do send entire static YML configuration in this case (parsed, so only recognised properties will be present in the payload), meaning that it will contain all pipeline definitions. The provider can remove or add pipeline definitions, so it has full control on the list we will show to the user.

Now, once user select a pipeline from the list in this dialog, we will send another request with trigger: manual, and we'll include a matching definition from the static YML configuration, if it is present. We will also send the selector of the pipeline definition the user has chosen in target.selector property.

Note that there might be no matched definition at all (in case the provider added completely new pipeline definition in the previous step) – I believe this is what you observed: pipelines_configuration.pipelines will be completely empty in this case. However, Dynamic Pipelines provider can recognise this case based on trigger and target.selector and return a configuration with pipeline definition that matches the selector and that we'll select and run.

Hope this helps. Let me know if you have any questions.

Cheers,
Daniil