nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How to reuse some steps across pipelines but with minor changes

nagarro_bhavya

I have a use case wherein my repository works on multiple AWS Lambda functions. I am using bitbucket pipelines to test each of those python functions (based on the changeset condition) with pytest and pylint and send an email in case any of those fail.

All the steps are exactly same for all the functions except some things such as

1. the path where to run pytest

2. the message to send in email etc.

My pipeline looks something like this:

- step:
  name: function 1
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_1
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_1/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 1 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_1/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/* 


- step:
  name: function 2
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_2
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_2/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 2 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_2/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/*

All the bold text is common to each step. Whereas only the italic part varies.

I have 10 such functions. I am looking a way out so that I dont have to repeat all these lines again and again. As you can see keeping two arguments which change with each step will solve the issue.

I cant use YAML anchors as there are a couple of changes here and there which are specific to each step.

Accepted answers

Theodora Boudale

Hi @nagarro_bhavya and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

https://jira.atlassian.com/browse/BCLOUD-20821

The way to reuse code in Pipelines at the moment is with YAML anchors, and for your specific use case, my suggestion would be a combination of deployment environments+variables along with YAML anchors.

You mentioned that you cannot use YAML anchors as there are a couple of changes that are specific to each step. Could you please give an example of what kind of changes? Do you want to execute extra commands for certain steps? Or something different?

Kind regards,
Theodora

All comments

Theodora Boudale

Hi @nagarro_bhavya and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

https://jira.atlassian.com/browse/BCLOUD-20821

Kind regards,
Theodora

nagarro_bhavya

Hello @Theodora Boudale ,

Thanks for your reply.

For example, in the above snippet, the underlined or non-bold text are the changes which are step-specific and hence using anchors will not be possible.

Theodora Boudale

Hi @nagarro_bhavya,

Thanks for getting back to me.

It should be possible to use YAML anchors in combination with deployment environments and deployment variables to reuse code.

What cannot be included in the YAML anchor is the step name and the condition.

It is not possible to assign a variable to a step's name, and a variable used in a path of includePaths: won't work because the condition is used before the build execution, while the variables are part of a container during its execution only.

The rest of the content though can go in a yaml anchor.

Please allow me to give you an example with 2 functions, to show how you could achieve reuse of code:

1. In Bitbucket website, open the repo where you have this pipeline, go to its Repository settings > Deployments and create one environment for each function. In my case, I created 2, and I named them Function_1 and Function_2.

For each one of these environments, create a variable with the name function, and assign it the respective value that corresponds to the name of the directory in your repo.

E.g. for the environment Function_1, I created a variable named function, with the value function_1.
For the environment Function_2, I created a variable named function, with the value function_2.

It's important that this variable has the same name for all environments you'll use for your functions.

2. The YAML file can look as follows:

definitions: 
  steps:
    - step: &myfunction
        caches:
          - pip
        script:
          - pip install --upgrade pip
          - other common steps
          - cd root_folder/$function
          - pytest
          - other common steps
        after-script:
          - echo $BITBUCKET_EXIT_CODE
          - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
          - pipe: atlassian/email-notify:0.4.4
            variables:
              USERNAME: some username
              PASSWORD: some password
              FROM: $FROM_EMAIL
              Other common variables
              SUBJECT: '${function} Failed'
              ATTACHMENTS: '/opt/atlassian/pipelines/agent/build/root_folder/${function}/coverage/index.html'
          - fi; 
        artifacts:
          - test-reports/*

pipelines:
  default:
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_1/**
              - utility/scripts/**
        name: function 1
        deployment: Function_1
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_2/**
              - utility/scripts/**
        name: function 2
        deployment: Function_2

The biggest part of your step, with the exception of the name and the condition, can go in a yaml anchor. Note the use of the variable ${function} in the yaml anchor.

Later, when we reference the YAML anchor in a specific step, we also use the keyword deployment with a certain environment.

For the first step that has deployment: Function_1, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_1 environment, which is function_1.

The second step has deployment: Function_2, so when the script runs, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_2 environment, which is function_2.

We need to give the name of the step outside the anchor, as we can't assign variables to a step's name.
Additionally, the condition: / changesets: / includePaths: must also be outside the anchor, as we cannot use a variable's name in the path. E.g. if we had something like - root_folder/$function/** in a condition inside a step, it would not work.

So, even though certain things need to be repeated, you can reuse your script and after-script.

Is this something that would work for you?

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

nagarro_bhavya

Thank you @Theodora Boudale

This answers my question,

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well).

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bitbucket

bitbucket-cloud

bitbucket-pipelines

atl-orange

continuous-delivery