How can I define reusable scripts and run them in parallel for a branch

[Eric Poe]

Given the following pipeline configuration, the Pipeline Validator claims it is good but the pipeline gives an error: `[pipelines > branches > FOO-10751-snapshot-used-as-clean-source > 0 > parallel > steps > 0]. The step section is empty or null.`

• Am I wrong? How can I fix this?
• Is the pipeline runner wrong?
• Is the Pipeline Validator wrong?

definitions:
  steps: # Note: the "&" is a yaml anchor that allows reusing this particular step
    - step: 
        name: &PHP-Code-Quality
        image: foo/php-ci:1
        script:
          - |
            php bunch-of-stuff.php
    - step:
        name: &JS-Code-Lint
        image: node:18
        script:
          - |
            npx bunch-of-stuff
    - step:
        name: &Update-Snapshot
        runs-on:
          - server.test
          - self.hosted
          - linux.shell
        script:
          - bunch-of-stuff.sh
pipelines:
  default:
    - parallel:
        steps:
          - step: *PHP-Code-Quality
          - step: *JS-Code-Lint
          - step: *Update-Snapshot
  branches:
    FOO-10751-snapshot-used-as-clean-source:
      - parallel:
          steps:
            - step: *PHP-Code-Quality
            - step: *JS-Code-Lint
            - step: *Update-Snapshot

Eventually, I will want to use a specific branch for staging a release via Pipelines. This is a preliminary step towards that goal.

[Theodora Boudale · Accepted solution]

Hi Eric and welcome to the community!

There is a bug with the validator and it doesn't show any errors when yaml anchors are used.

I think the culprit for the error you see is that you are using the anchors in the step name, instead of after - step:

Can you try adjusting the definitions as follows, and let me know if it works?

definitions:
  steps: # Note: the "&" is a yaml anchor that allows reusing this particular step
    - step: &PHP-Code-Quality
        name: PHP-Code-Quality
        image: foo/php-ci:1
        script:
          - |
            php bunch-of-stuff.php
    - step: &JS-Code-Lint
        name: JS-Code-Lint
        image: node:18
        script:
          - |
            npx bunch-of-stuff
    - step: &Update-Snapshot
        name: Update-Snapshot
        runs-on:
          - server.test
          - self.hosted
          - linux.shell
        script:
          - bunch-of-stuff.sh

Kind regards,
Theodora

[Eric Poe]

That was it. Thank you!

[Theodora Boudale]

You are very welcome! Please feel free to reach out if you ever need anything else!

[Peter Falck Grony]

Hello Eric,

I didn't know about the Pipeline Validator, so I can't speak to who is wrong between that and the runner. However, I can share that the pipelines in my company looks like this to re-use steps:

aliases:
  - &build_and_test
    name: Build and test
    image: osrf/ros:noetic-desktop-full-bionic
    script:
    - apt update -y
    - apt install git ssh -y
    - git clone git@bitbucket.org:<redacted>/bitbucket-ros.git
    - bitbucket-ros/prepare-workspace.bash
    - bitbucket-ros/build.bash
    - bitbucket-ros/test.bash

  - &deploy
    name: Deploy
    image: osrf/ros:noetic-desktop-full-bionic
    deployment: production
    artifacts:
      - "*.deb"
    script:
    - apt update -y
    - apt install git ssh -y
    - git clone git@bitbucket.org:<redacted>/bitbucket-ros.git
    - bitbucket-ros/binarize.bash
    - bitbucket-ros/upload.bash
    - bitbucket-ros/docs.bash
    - bitbucket-ros/trigger-upstream-projects.py

pipelines:
  default:
    - step:
        <<: *build_and_test
    - step:
        <<: *deploy
        trigger: manual

  pull-requests:
    '**':
      - step:
          <<: *build_and_test

  tags:
     '**':
      - step:
          <<: *build_and_test
      - step:
          <<: *deploy

Biggest difference seems to be that we have `aliases` instead of definitions at the top and we use `<<: *anchor` to get the anchor. 

Note: I have found, when trying things out I found on the internet, that Bitbucket is not following standards for Markdown and Yaml - It's a mess. And Yaml in itself is also a mess... You may enjoy reading: The yaml document from hell 

[Eric Poe]

Nice. I'm not familiar with `<<:` so I'll look into it.