How to reuse some steps across pipelines but with minor changes

[nagarro_bhavya]

I have a use case wherein my repository works on multiple AWS Lambda functions. I am using bitbucket pipelines to test each of those python functions (based on the changeset condition) with pytest and pylint and send an email in case any of those fail.

All the steps are exactly same for all the functions except some things such as

1. the path where to run pytest

2. the message to send in email etc.

My pipeline looks something like this:

- step:
  name: function 1
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_1
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_1/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 1 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_1/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/* 


- step:
  name: function 2
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_2
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_2/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 2 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_2/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/*

All the bold text is common to each step. Whereas only the italic part varies.

I have 10 such functions. I am looking a way out so that I dont have to repeat all these lines again and again. As you can see keeping two arguments which change with each step will solve the issue.

I cant use YAML anchors as there are a couple of changes here and there which are specific to each step.

[Theodora Boudale · Accepted solution]

Hi @nagarro_bhavya and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

• https://jira.atlassian.com/browse/BCLOUD-20821

The way to reuse code in Pipelines at the moment is with YAML anchors, and for your specific use case, my suggestion would be a combination of deployment environments+variables along with YAML anchors.

You mentioned that you cannot use YAML anchors as there are a couple of changes that are specific to each step. Could you please give an example of what kind of changes? Do you want to execute extra commands for certain steps? Or something different?

Kind regards,
Theodora

[nagarro_bhavya]

Hello @Theodora Boudale ,

Thanks for your reply. 

For example, in the above snippet, the underlined or non-bold text are the changes which are step-specific and hence using anchors will not be possible.

[Theodora Boudale]

Hi @nagarro_bhavya,

Thanks for getting back to me.

It should be possible to use YAML anchors in combination with deployment environments and deployment variables to reuse code.

What cannot be included in the YAML anchor is the step name and the condition.

It is not possible to assign a variable to a step's name, and a variable used in a path of includePaths: won't work because the condition is used before the build execution, while the variables are part of a container during its execution only.

The rest of the content though can go in a yaml anchor.

Please allow me to give you an example with 2 functions, to show how you could achieve reuse of code:

1. In Bitbucket website, open the repo where you have this pipeline, go to its Repository settings > Deployments and create one environment for each function. In my case, I created 2, and I named them Function_1 and Function_2.

For each one of these environments, create a variable with the name function, and assign it the respective value that corresponds to the name of the directory in your repo.

E.g. for the environment Function_1, I created a variable named function, with the value function_1.
For the environment Function_2, I created a variable named function, with the value function_2.

It's important that this variable has the same name for all environments you'll use for your functions.

2. The YAML file can look as follows:

definitions: 
  steps:
    - step: &myfunction
        caches:
          - pip
        script:
          - pip install --upgrade pip
          - other common steps
          - cd root_folder/$function
          - pytest
          - other common steps
        after-script:
          - echo $BITBUCKET_EXIT_CODE
          - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
          - pipe: atlassian/email-notify:0.4.4
            variables:
              USERNAME: some username
              PASSWORD: some password
              FROM: $FROM_EMAIL
              Other common variables
              SUBJECT: '${function} Failed'
              ATTACHMENTS: '/opt/atlassian/pipelines/agent/build/root_folder/${function}/coverage/index.html'
          - fi; 
        artifacts:
          - test-reports/*

pipelines:
  default:
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_1/**
              - utility/scripts/**
        name: function 1
        deployment: Function_1
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_2/**
              - utility/scripts/**
        name: function 2
        deployment: Function_2

The biggest part of your step, with the exception of the name and the condition, can go in a yaml anchor. Note the use of the variable ${function} in the yaml anchor.

Later, when we reference the YAML anchor in a specific step, we also use the keyword deployment with a certain environment.

For the first step that has deployment: Function_1, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_1 environment, which is function_1.

The second step has deployment: Function_2, so when the script runs, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_2 environment, which is function_2.

We need to give the name of the step outside the anchor, as we can't assign variables to a step's name.
Additionally, the condition: / changesets: / includePaths: must also be outside the anchor, as we cannot use a variable's name in the path. E.g. if we had something like - root_folder/$function/** in a condition inside a step, it would not work.

So, even though certain things need to be repeated, you can reuse your script and after-script.

Is this something that would work for you?

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

[nagarro_bhavya]

Thank you @Theodora Boudale

This answers my question,

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well).

[Theodora Boudale]

Hi @nagarro_bhavya,

You are very welcome and I apologize for my late reply, it looks that I've been missing some notifications.

Regarding your last question:

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well)

If you have the premium plan, you can restrict which branches the deployment can run for, e.g. the deployment can run only for master branch or development branch, if there are new commits to this branch. But then, the deployment needs to be specified for a branch in the yml file.

If you want to use the deployments in the 'pull-requests:' definition of the yml file, then you specify in the yml file the source branch (not the destination branch) and the builds run when you create the PR (not when it is merged to the destination branch). It is not possible to specify a restriction for pull-requests builds based on the destination branch, if this is your use case, so you wouldn't need the premium plan in this case.

Please feel free to let me know if you have any further questions.

Kind regards,
Theodora

[nagarro_bhavya]

Hey @Theodora Boudale 

I just read 12 days of CI/CD series and came across the concept of Default Variables as mentioned here: https://bitbucket.org/blog/updates-to-bitbucket-pipes

Can you please provide an example of how can I use it. As you see, in my case, I have to use a pipe multiple times which has 3-4 variables in common. Hence, it will help me a lot.

[Alexander Schomburg]

Before anyone else spends time in testing this... I did, and I'm fairly confident that the mentioned `${function}` for deployments is not supported (anymore?) Too bad because to my knowledge Bitbucket Cloud does not offer any alternative to it.

[Theodora Boudale]

Hi Alexander,

Using a variable with the deployment keyword, something like in the example below, is not supported at the moment.

- step:
    name: Deploy to production
    deployment: ${function}
    script:
      - echo "Deploy to production"

We have a feature request about this here: https://jira.atlassian.com/browse/BCLOUD-19611

Is this what you are referring to or are you referring to something else in the example I provided?

Kind regards,
Theodora

[Alexander Schomburg]

Hi Theodora,

Yes, I was referring to using the value of the deployment keyword inside a YAML anchor through `${function}`. I read your comment from July 30, 2021 that suggested that this functionality is supported, but found that it's not working. To stop others wasting their time in trying, I made my comment.

Thanks for your confirmation. I hope that this functionality will be introduced in the future.

Best regards

Alex