Introducing parent/child pipelines

[#1]

We’re excited to announce the launch of parent/child pipelines for Bitbucket Pipelines.

This powerful new capability lets you define a step within a pipeline that triggers and encapsulates a whole other pipeline, which can help to streamline more complex workflows into modular pieces and achieve greater parallelism within your pipeline.

Check out the release blog to learn more.

[Chris Callahan]

I've attempted migrate one of our repos that currently rely on the trigger pipe to this but am running into an issue or undocumented limitation. It doesn't appear the parent step that runs the child pipeline supports conditions. This is causing the child pipelines to run regardless of condition logic for the step.

[Edmund Munday]

@Chris Callahan - thanks for your feedback, checking this with the team now - we'll have an update for you in the next 24hrs.

[Edmund Munday]

Update: We've identified the issue causing this behaviour and will have a fix deployed by EOD tomorrow. Depending on your geo, I'd suggest giving your workflow another try on Friday. Thanks again @Chris Callahan

[Ori Saporta]

A couple of questions arise regarding pipeline variables. 
Is there a way to send specific values directly to the child pipeline?
Are they inherited from the parent pipeline?

[matteo_castellotti]

I'm trying out this new feature, and it seems like artifacts exported in a parent step aren't being passed to the child pipelines. Do you know if this is the expected outcome?

[Mike Turalenka]

Thank you for the new feature, but moving logic from definitions to multiple custom pipelines will result in larger amount of pipelines listed in "Run Pipeline" dropdown, when not necessary all of them were added for public usage, but only for shared/reusable steps. Will it be possible to add an attribute to the custom pipeline to define either it has to be visible in UI or not? like `public: true|false`

[Edmund Munday]

@Ori Saporta - great questions:

As of today, child pipeline are entirely encapsulated and can't consume context from the parent, just the repository and workspace.

However, context sharing from parent to child is literally feature #1 on our "coming next" list and we will be making that available soon. This will work very similarly to how variables are passed into Pipes today.

It will also support runtime generated variables created via exporting values from earlier steps in your Pipeline.

[Edmund Munday]

@matteo_castellotti - that is the expected behaviour for the moment, child pipelines are currently quite encapsulated and do not share much context from the parent. More advanced context sharing is the first thing we're working on post-release.

Just for our reference, would you be able to share an example of the use-case you're trying to achieve where you'd like to have shared artifacts between parent & child?

[Edmund Munday]

@Mike Turalenka - that's a really excellent point, thanks for raising this. We're going to have a look into something along the lines of what you suggested. Can't promise anything right now, but you raise a really good point, so hopefully there's something nice and simple we can do to prevent clogging up your UI.

[matteo_castellotti]

On our side, we have something like the pipeline that I shared: a parent pipeline for the dev branch. As a first step, we build the app and export some artifacts that we need to pass into all the test steps that we want to call as child pipelines.

Both of these child pipelines will run in parallel our tests.

Do you need more information?

pipelines:
   branches:
      'dev':
         - step:
            name: Build
            script:
               - ....
            artifacts:
               - artifact-1
               - artifact-2
         - step:
            type: pipeline
            environment: my-env
            name: Unit tests
            custom: unit-tests-pipeline
         - step:
            type: pipeline
            environment: my-env
            name: It tests
            custom: it-tests-pipeline

custom:

   unit-tests-pipeline:
      - parallel:
         - step: *unit-test1
         - step: *unit-test2
         - step: *unit-test3

   it-tests-pipeline:
      - parallel:
         - step: *it-test1
         - step: *it-test2
         - step: *it-test3

[Chris Callahan]

@Edmund Munday Conditions are now working as expected, thanks!

I know there are some efforts to refactor "Deployments", but are you able to provide the reasoning behind not allowing the use of Deployments in child pipelines?

[Yash Smith]

Parent/child pipelines in CI/CD (like GitLab CI) allow you to break a complex pipeline into smaller, reusable components. A parent pipeline triggers one or more child pipelines, improving modularity, reuse, and maintainability. This setup helps in managing large projects by splitting responsibilities across pipeline files while maintaining control flow.

[Chris Callahan]

@Edmund Munday The conditions issue has returned again, it is no longer working and firing all child steps regardless of condition.

Update: This is now working as expected again after the weekend.

[Radu Cristescu]

Let's see if I understand this feature in comparison to triggered pipelines.

The jist

It's not an in-place replacement for triggered pipelines.

My understanding

These pipelines are run as if they're in the repository that calls them (except they can't receive any context yet, making them unable to use variables), while triggered pipelines run in the repository they reside in.

The child pipelines in this feature are defined in the same repository, not in a separate repository like triggered pipelines.

We cannot yet pass variables to the child pipeline.

We cannot use the "Deployment" concept with parent+child pipelines. It makes sense to not allow "deployment" for the child pipeline, but why the parent? I'm not familiar with Environments, but Deployments are useful for quickly rolling back buggy releases.

Use cases

I run an Ansible playbook in a custom pipeline in a specialised repository. The Ansible configuration does not reside in the pipeline, but in the repository itself as a collection of many files. I do not want to add these files to repositories that use this pipeline.

In another instance, I have E2E integration tests in another repository. I currently trigger those pipelines and use build minutes polling.

I currently work around this by having several self-hosted runners on a small computer just for the purpose of polling triggered pipelines. They don't use much memory, so I'm skirting the memory recommendations for self-hosted runners and overprovisioned it with more runners than recommended.

[gokhanozgozen]

Is there a built-in mechanism to prevent circular pipeline calls?

X -> Y -> Z -> X.

[Chris Callahan]

@gokhanozgozen A child pipeline cannot contain calls to additional child pipelines so this shouldn't be possible.

[Anton kristensen]

Will there be a video tutorial out about this? I belive it to be quite difficult to understand the reason for use. I get that a pipeline can call another pipeline, but if that pipeline is encapsulated and just runs parallel to the parent, then the only benefit is that it runs parallel and breaks a monolithic block of steps. I might be understanding this completely wrong, and if that's so excuse me. But anyways a video tutorial would be great.

Also, wouldn't it be great if there was some way if triggering to different branches. For example this:

pipelines:
    default:
      - step: *initialize-step
      - step:
           type: pipeline
           name: "Trigger branch child"
           custom: test-ci-cd-pipeline
           trigger:
               branches:
                     test-ci-cd

custom:
     test-ci-cd-pipeline:
          - step: *build-test-step

[Jur]

Why is a pipeline reference that is exported always using the environment from the repo that imports it? Can we make it an option to use the environment thats in the source repo where the pipeline is exported? I would really like that. Now I still can't use this because I need environment variables from the source repo with this pipeline 

[Casper Rademaker]

@Edmund Munday Great new feature! We already started using it.

However, it seems that the Bitbucket pipelines schema published at api.bitbucket.org/schemas/pipelines-configuration (or shipped with the Atlassian VS Code extension) isn't updated yet with this new feature, causing schema validation to fail whenever we do something like this:

pipelines:

  custom:

    generic-pr-pipeline:

      import: bitbucket-pipeline-templates:main:generic-pr-pipeline

  pull-requests:

    "**":

      - step:

          type: pipeline

          name: "Generic PR pipeline"

          custom: generic-pr-pipeline

[Stefan Schindler]

Does this make it possible to stop the parallel jobs when one fails?

[Radu Cristescu]

Since parent/child pipelines don't support the "deployment" keyword, what's the alternative process for tracking deployments and running previous deployments (to revert changes before new source code is ready) or keep track of when a deployment was made? The "environment" keyword appears to just provide and environment to the pipeline, with no deployment tracking.

[Nitin Choudhary]

I moved our steps into child pipelines, but their logs no longer appear on the parent pipeline page. The parent pipeline also shows misleading build minutes since it excludes child pipelines. I expected more parallelism as described in the blog post, but build times have not improved. Instead, it added extra steps to check logs in child pipelines and made the parent pipeline metrics less accurate.

[Stanislau Yarotski]

It would be great to be able to share artifacts from child pipelines into the parent pipeline.

[Alex Fok]

Also sharing artifacts from parent pipeline to child pipelines as well.