Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to reuse some steps across pipelines but with minor changes

Edited

I have a use case wherein my repository works on multiple AWS Lambda functions. I am using bitbucket pipelines to test each of those python functions (based on the changeset condition) with pytest and pylint and send an email in case any of those fail.

 

All the steps are exactly same for all the functions except some things such as

1. the path where to run pytest

2. the message to send in email etc.

 

My pipeline looks something like this:

- step:
  name: function 1
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_1
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_1/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 1 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_1/coverage/index.html'
     - fi;
   artifacts:
     - test-reports/*


- step:
  name: function 2
  caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
    - cd root_folder/function_2
    - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
        - root_folder/function_2/**
        - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 2 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_2/coverage/index.html'
     - fi;
   artifacts:
     - test-reports/*

All the bold text is common to each step. Whereas only the italic part varies.

I have 10 such functions. I am looking a way out so that I dont have to repeat all these lines again and again. As you can see keeping two arguments which change with each step will solve the issue.

I cant use YAML anchors as there are a couple of changes here and there which are specific to each step.

1 answer

1 accepted

0 votes
Answer accepted

Hi @nagarro_bhavya and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

The way to reuse code in Pipelines at the moment is with YAML anchors, and for your specific use case, my suggestion would be a combination of deployment environments+variables along with YAML anchors.

You mentioned that you cannot use YAML anchors as there are a couple of changes that are specific to each step. Could you please give an example of what kind of changes? Do you want to execute extra commands for certain steps? Or something different?

Kind regards,
Theodora

Hello @Theodora Boudale ,

Thanks for your reply. 

For example, in the above snippet, the underlined or non-bold text are the changes which are step-specific and hence using anchors will not be possible.

Hi @nagarro_bhavya,

Thanks for getting back to me.

It should be possible to use YAML anchors in combination with deployment environments and deployment variables to reuse code.

What cannot be included in the YAML anchor is the step name and the condition.

It is not possible to assign a variable to a step's name, and a variable used in a path of includePaths: won't work because the condition is used before the build execution, while the variables are part of a container during its execution only.

The rest of the content though can go in a yaml anchor.

Please allow me to give you an example with 2 functions, to show how you could achieve reuse of code:

 

1. In Bitbucket website, open the repo where you have this pipeline, go to its Repository settings > Deployments and create one environment for each function. In my case, I created 2, and I named them Function_1 and Function_2.

For each one of these environments, create a variable with the name function, and assign it the respective value that corresponds to the name of the directory in your repo.

E.g. for the environment Function_1, I created a variable named function, with the value function_1.
For the environment Function_2, I created a variable named function, with the value function_2.

It's important that this variable has the same name for all environments you'll use for your functions.

2. The YAML file can look as follows:

definitions: 
steps:
- step: &myfunction
caches:
- pip
script:
- pip install --upgrade pip
- other common steps
- cd root_folder/$function
- pytest
- other common steps
after-script:
- echo $BITBUCKET_EXIT_CODE
- if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed";
- pipe: atlassian/email-notify:0.4.4
variables:
USERNAME: some username
PASSWORD: some password
FROM: $FROM_EMAIL
Other common variables
SUBJECT: '${function} Failed'
ATTACHMENTS: '/opt/atlassian/pipelines/agent/build/root_folder/${function}/coverage/index.html'
- fi;
artifacts:
- test-reports/*

pipelines:
default:
- step:
<<: *myfunction
condition:
changesets:
includePaths:
- root_folder/function_1/**
- utility/scripts/**
name: function 1
deployment: Function_1
- step:
<<: *myfunction
condition:
changesets:
includePaths:
- root_folder/function_2/**
- utility/scripts/**
name: function 2
deployment: Function_2

The biggest part of your step, with the exception of the name and the condition, can go in a yaml anchor. Note the use of the variable ${function} in the yaml anchor.

Later, when we reference the YAML anchor in a specific step, we also use the keyword deployment with a certain environment.

For the first step that has deployment: Function_1, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_1 environment, which is function_1.

The second step has deployment: Function_2, so when the script runs, the variable ${function} in the script will take the value of the deployment variable function that we defined in Function_2 environment, which is function_2.

We need to give the name of the step outside the anchor, as we can't assign variables to a step's name.
Additionally, the condition: / changesets: / includePaths: must also be outside the anchor, as we cannot use a variable's name in the path. E.g. if we had something like - root_folder/$function/** in a condition inside a step, it would not work.

So, even though certain things need to be repeated, you can reuse your script and after-script.

Is this something that would work for you?

Please feel free to let me know if you have any questions.

Kind regards,
Theodora

Like nagarro_bhavya likes this

Thank you @Theodora Boudale

This answers my question,

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well).

Hi @nagarro_bhavya,

You are very welcome and I apologize for my late reply, it looks that I've been missing some notifications.

Regarding your last question:

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well)


If you have the premium plan, you can restrict which branches the deployment can run for, e.g. the deployment can run only for master branch or development branch, if there are new commits to this branch. But then, the deployment needs to be specified for a branch in the yml file.

 

If you want to use the deployments in the 'pull-requests:' definition of the yml file, then you specify in the yml file the source branch (not the destination branch) and the builds run when you create the PR (not when it is merged to the destination branch). It is not possible to specify a restriction for pull-requests builds based on the destination branch, if this is your use case, so you wouldn't need the premium plan in this case.

Please feel free to let me know if you have any further questions.

Kind regards,
Theodora

Hey @Theodora Boudale 

I just read 12 days of CI/CD series and came across the concept of Default Variables as mentioned here: https://bitbucket.org/blog/updates-to-bitbucket-pipes

Can you please provide an example of how can I use it. As you see, in my case, I have to use a pipe multiple times which has 3-4 variables in common. Hence, it will help me a lot.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
TAGS
Community showcase
Published in Bitbucket

📣 Calling Bitbucket Data Center customers to participate in research

Hi everyone, Are you Bitbucket DC customer? If so, we'd love to talk to you! Our team wants to dive deep to understand your long-term plans regarding Bitbucket DC and Atlassian Cloud. Do you plan...

226 views 2 5
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you