How to reuse some steps across pipelines but with minor changes

nagarro_bhavya July 20, 2021

I have a use case wherein my repository works on multiple AWS Lambda functions. I am using bitbucket pipelines to test each of those python functions (based on the changeset condition) with pytest and pylint and send an email in case any of those fail.

 

All the steps are exactly same for all the functions except some things such as

1. the path where to run pytest

2. the message to send in email etc.

 

My pipeline looks something like this:

- step:
  name: function 1
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_1
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_1/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 1 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_1/coverage/index.html'
     - fi;
   artifacts:
     - test-reports/*


- step:
  name: function 2
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_2
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_2/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 2 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_2/coverage/index.html'
     - fi;
   artifacts:
     - test-reports/*

All the bold text is common to each step. Whereas only the italic part varies.

I have 10 such functions. I am looking a way out so that I dont have to repeat all these lines again and again. As you can see keeping two arguments which change with each step will solve the issue.

I cant use YAML anchors as there are a couple of changes here and there which are specific to each step.

1 answer

1 accepted

0 votes
Answer accepted
Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 23, 2021

Hi @nagarro_bhavya and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

The way to reuse code in Pipelines at the moment is with YAML anchors, and for your specific use case, my suggestion would be a combination of deployment environments+variables along with YAML anchors.

You mentioned that you cannot use YAML anchors as there are a couple of changes that are specific to each step. Could you please give an example of what kind of changes? Do you want to execute extra commands for certain steps? Or something different?

Kind regards,
Theodora

nagarro_bhavya July 28, 2021

Hello @Theodora Boudale ,

Thanks for your reply. 

For example, in the above snippet, the underlined or non-bold text are the changes which are step-specific and hence using anchors will not be possible.

Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 30, 2021

Hi @nagarro_bhavya,

Thanks for getting back to me.

It should be possible to use YAML anchors in combination with deployment environments and deployment variables to reuse code.

What cannot be included in the YAML anchor is the step name and the condition.

It is not possible to assign a variable to a step's name, and a variable used in a path of includePaths: won't work because the condition is used before the build execution, while the variables are part of a container during its execution only.

The rest of the content though can go in a yaml anchor.

Please allow me to give you an example with 2 functions, to show how you could achieve reuse of code:

 

1. In Bitbucket website, open the repo where you have this pipeline, go to its Repository settings > Deployments and create one environment for each function. In my case, I created 2, and I named them Function_1 and Function_2.

For each one of these environments, create a variable with the name function, and assign it the respective value that corresponds to the name of the directory in your repo.

E.g. for the environment Function_1, I created a variable named function, with the value function_1.
For the environment Function_2, I created a variable named function, with the value function_2.

It's important that this variable has the same name for all environments you'll use for your functions.

2. The YAML file can look as follows:

definitions: 
steps:
- step: &myfunction
caches:
- pip
script:
- pip install --upgrade pip
- other common steps
- cd root_folder/$function
- pytest
- other common steps
after-script:
- echo $BITBUCKET_EXIT_CODE
- if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
- pipe: atlassian/email-notify:0.4.4
variables:
USERNAME: some username
PASSWORD: some password
FROM: $FROM_EMAIL
Other common variables
SUBJECT: '${function} Failed'
ATTACHMENTS: '/opt/atlassian/pipelines/agent/build/root_folder/${function}/coverage/index.html'
- fi;
artifacts:
- test-reports/*

pipelines:
default:
- step:
<<: *myfunction
condition:
changesets:
includePaths:
- root_folder/function_1/**
- utility/scripts/**
name: function 1
deployment: Function_1
- step:
<<: *myfunction
condition:
changesets:
includePaths:
- root_folder/function_2/**
- utility/scripts/**
name: function 2
deployment: Function_2

The biggest part of your step, with the exception of the name and the condition, can go in a yaml anchor. Note the use of the variable ${function} in the yaml anchor.

Later, when we reference the YAML anchor in a specific step, we also use the keyword deployment with a certain environment.

For the first step that has deployment: Function_1, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_1 environment, which is function_1.

The second step has deployment: Function_2, so when the script runs, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_2 environment, which is function_2.

We need to give the name of the step outside the anchor, as we can't assign variables to a step's name.
Additionally, the condition: / changesets: / includePaths: must also be outside the anchor, as we cannot use a variable's name in the path. E.g. if we had something like - root_folder/$function/** in a condition inside a step, it would not work.

So, even though certain things need to be repeated, you can reuse your script and after-script.

Is this something that would work for you?

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

Like nagarro_bhavya likes this
nagarro_bhavya July 31, 2021

Thank you @Theodora Boudale

This answers my question,

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well).

Theodora Boudale
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
August 24, 2021

Hi @nagarro_bhavya,

You are very welcome and I apologize for my late reply, it looks that I've been missing some notifications.

Regarding your last question:

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well)


If you have the premium plan, you can restrict which branches the deployment can run for, e.g. the deployment can run only for master branch or development branch, if there are new commits to this branch. But then, the deployment needs to be specified for a branch in the yml file.

 

If you want to use the deployments in the 'pull-requests:' definition of the yml file, then you specify in the yml file the source branch (not the destination branch) and the builds run when you create the PR (not when it is merged to the destination branch). It is not possible to specify a restriction for pull-requests builds based on the destination branch, if this is your use case, so you wouldn't need the premium plan in this case.

Please feel free to let me know if you have any further questions.

Kind regards,
Theodora

nagarro_bhavya August 31, 2021

Hey @Theodora Boudale 

I just read 12 days of CI/CD series and came across the concept of Default Variables as mentioned here: https://bitbucket.org/blog/updates-to-bitbucket-pipes

Can you please provide an example of how can I use it. As you see, in my case, I have to use a pipe multiple times which has 3-4 variables in common. Hence, it will help me a lot.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
AUG Leaders

Atlassian Community Events