Bitbucket pipeline reuse of code

[Samuel]

I have lot's of duplicate code in my pipeline yaml. I tried the 'YAML anchors and references' approach suggested in Q&A https://community.atlassian.com/t5/Bitbucket-questions/How-to-reuse-steps-for-different-branches/qaq-p/875615 but it doesn't work and I get an error when I want to launch my pipeline.

The error is sometling like:

Configuration error

There is an error in your bitbucket-pipelines.yml at [pipelines > custom > build-internal > 0 > step > script > 1]. To be precise: Missing or empty command string. Each item in this list should either be a single command string or a map defining a pipe invocation.

Validate your bitbucket-pipelines.ymlView bitbucket-pipelines.yml

Below is a code sample to test:

image: maven

clone:
 depth: 500

commonStep: &commonStep 
 - step:
  script:
   - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
 custom:
  build-internal:
   - step:
    name: Build maven internal
    caches:
     - maven
    script:
     - echo "Lot's of code redundancy"
     - <<: *commonStep

  deploy-internal-tst:
   - step:
    name: Deploy to test
    deployment: test
    script:
     - echo "Lot's of code redundancy"
     - <<: *commonStep

What is going wrong?

[Daniel Santos · Accepted solution]

Hi @Samuel

I think the problem you are facing is caused by indentation. The common step behaves like a full step, not only an extra command in the script section. 

I understand you when you say that there is no validation error in the syntax but there are compilation errors. This is caused by a problem to validate files using anchors. This problem is already reported here:

• Better YML validation when configuration uses YML anchors and references

I've run this test and it worked as expected. Please notice that I used the default section to simplify running it locally.

image: maven

clone:
  depth: 500

commonStep: &commonStep
  step:
    script:
      - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
  default:
    - step:
        name: Build maven internal
        caches:
          - maven
        script:
          - echo "Lot's of code redundancy"
    - <<: *commonStep

You will see that

 - <<: *commonStep

is aligned with the above step and it will create an extra step when compiling.

I also built another 2 tests examples to show a possible way to use anchors.

The first one is similar to the one shared in that question you mentioned:

commonStep: &commonStep
  step:
    script:
      - echo "common step"

commonScript: &commonScript
  script:
    - echo "common script"

pipelines:
  default:
    - <<: *commonStep
    - step:
        <<: *commonScript

The second one is similar to what is shared in our docs:

definitions:
  step: &myCommonStep
    script: &myCommonScript
      - echo "definition command"

pipelines:
  default:
    - step: *myCommonStep
    - step:
        script: *myCommonScript
    - step:
        <<: *myCommonStep
        name: my new name

Reference: YAML anchors - Atlassian Documentation

I tested both of them.

I hope that helps.

[Samuel]

Thanks Daniel, your reply got me going.

Rewriting my example the now working version looks like this:

# Docker image
image: maven

clone:
 depth: 500

commonStep1: &commonStep1
 - step:
  name: Step 1a
  script:
   - echo "Lots of duplicate code"
   - echo "Let's see the file structure"
   - mkdir -p test
   - echo "." > test/test-file2.txt
   - ls -la test/
  artifacts:
   - test/*.txt
commonStep2: &commonStep2
 - step:
  name: Step 1b
  script:
   - echo "Even more duplicate code"
   - echo "Let's see the file structure"
   - mkdir -p test
   - ls -la test/
pipelines:
 custom:
  build-internal:
   - step:
    name: Step 1
    script:
     - echo "Let's get started"
     - mkdir -p test
     - echo "." > test/test-file1.txt
     - ls -la test/
 
   - <<: *commonStep1
   - <<: *commonStep2

[Daniel Santos]

Hi @Samuel

I'm glad you were already finding your own way!
I wanted to share those examples as a reference for other users as well.

Thank you for taking the time to create this question and following up on it, I appreciate that.

Have a good one!

[rgbp]

Hi @Daniel Santos ,

I am trying to reuse a common script and add more script commands on the same step, but I am not able to do it. Is it possible? Here's what I am trying to do, from your example:

definitions:
  scripts: 
    script: &commonScript
      - echo "common script"

pipelines:
  default:
    - step:
        script:
          <<: *commonScript
          - echo "another script"

[Diônitas Mendes dos Santos]

@rgbp

I was having the same problem, this solves for me. take a try.

definitions:
  script: &commonScript
    echo "common script"

pipelines:
  default:
    - step:
        script:
          - *commonScript
          - echo "another script"

[Khalifa EL BANAN]

@Diônitas Mendes dos Santos I tried your solution after a plenty of tries but I get a configuration error at the beginning of the pipeline. Even if BB validator says that everything is fine with my pipeline.

PS: I'm using parallel steps

[raymond Kelly]

 @Khalifa EL BANAN  can you provide an example of your pipeline as I got it working. Note: I did see an error when I tried to make the reusable script a list and instead needed to make them a string and use &&

[Khalifa EL BANAN]

Yes sure, in general my pipeline looks like the following:

definitions:
  script: &myCommonScript
    - first script
    - second script

pipelines:
  pull-requests:
    feature/*:
      - step:
          name: 'first step'
          script:
            - some script

      - parallel:
        - step:
          name: 'parallel step 1'
          script:
            - *myCommonScript
            - a further script
        - step:
          name: 'parallel step 2'
          script:
            - *myCommonScript
            - another script

[raymond Kelly]

The issue is likely that 

&myCommonScript

is a list. When you try to use that in a script such as

        - step:
          name: 'parallel step 1'
          script:
            - *myCommonScript
            - a further script

what you actually have is a list of scripts within a script. My recommendation would instead be changing your pipeline to have the common script be a single operation such as

definitions:
  script: &myCommonScript
    first script &&
    second script

pipelines:
  pull-requests:
    feature/*:
      - step:
          name: 'first step'
          script:
            - some script

      - parallel:
        - step:
          name: 'parallel step 1'
          script:
            - *myCommonScript
            - a further script
        - step:
          name: 'parallel step 2'
          script:
            - *myCommonScript
            - another script

 There may be a cleaner way of doing this but I have not found it

[Khalifa EL BANAN]

thanks @raymond Kelly 

[Jim Fang]

It doesn't work. 

Configuration error

There is an error in your bitbucket-pipelines.yml at [pipelines > branches > TEST-DEV > 0 > step > script > 0]. To be precise: This section should be a string or a map (it is currently defined as a list).

[Diônitas Mendes dos Santos]

Hi Jim, note in my case a not using a list, is just a string. if you need more than one script in commonScript, you can do some thing like that

definitions:
  script: &commonScript
    echo "common script 1";
    echo "common script 2";


pipelines:
  default:
    - step:
        script:
          - *commonScript
          - echo "another script"

the yaml parse will treat as if it were a big string

echo "common script 1";echo "common script 2";

[Jim Fang]

Thanks @Diônitas Mendes dos Santos

i will post my bitbucket-pipelines.yml.  I check with bitbucket validatator. it said "valid", but i tried it with bitbucket pipeline. It reported error.

--------------------

There is an error in your bitbucket-pipelines.yml at [pipelines > branches > TEST-DEV > 0 > step > script > 0]. To be precise: This section should be a string or a map (it is currently defined as a list).

-------------------

```
image : nikolaik/python-nodejs:python3.8-nodejs15

definitions:
step: &myCommonStep
script: &myCommonScript
- curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
- unzip awscliv2.zip
- ./aws/install
- ls -lrta
- npm install -g serverless
- python --version
- serverless --version
- aws --version
- export roleARN=$ROLE_ARN_DEV
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID --profile loop-role
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY --profile loop-role
- echo "role_arn=$roleARN" >> ~/.aws/credentials
- echo "source_profile=default" >> ~/.aws/credentials
- cat ~/.aws/credentials
- aws configure set source_profile default --profile loop-role
- aws configure set role_arn $roleARN --profile loop-role
- aws configure set region ap-southeast-2 --profile loop-role
- aws configure set output json --profile loop-role
- aws configure list
- cat ~/.aws/credentials
- cat ~/.aws/config

pipelines:
branches:
TEST-DEV:
- step:
name: serverless build
caches:
- node
script:
- *myCommonScript
- npm install -g serverless-pseudo-parameters
- npm install -g serverless-python-requirements
- serverless deploy --stage dev --verbose --aws-profile loop-role

```

[Jim Fang]

# I tried to use code block, but after copy-paste, it disappear.
# This pipeline is to use serverless and assume role to deploy an application

Thanks 

[Vytenis Ščiukas]

For anyone that wants undocumented syntax for multiple scripts here is an example:

image: somerunnerimage:latest

definitions:
  scripts:
    - script: &setup-private-key
                echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
                chmod 600 ~/.ssh/id_rsa
    - script: &setup-environment
                export ENVIRONMENT=${BITBUCKET_DEPLOYMENT_ENVIRONMENT}
  steps:
    - step: &testing
        name: Testing
        runs-on:
          - "self.hosted"
        caches:
          - node
          - composer
        script:
          - phing -Denvironment=test
          - php artisan test --testsuite=Unit
  
    - step: &deployment
        name: Building & deploying
        runs-on:
          - "self.hosted"
        caches:
          - node
          - composer
        script:
          - *setup-private-key
          - *setup-environment
          - phing -Denvironment=${ENVIRONMENT}
          - dep deploy --revision=${BITBUCKET_TAG}

pipelines:
  tags:
    r[0-9].[0-9].[0-9]:
      - step:
          <<: *testing
      - step:
          <<: *deployment
          deployment: production
  branches:    
    master:
      - step:
          <<: *deployment
          deployment: staging

[matteo_migliaccio]

I'm sure we're not alone wondering why this is still undocumented, where can we find more about this and, more importantly, if there's a log somewhere we can monitor to be up to date with new bb pipelines features, we've stumble on this comment just by chance.

[Ekaterina Khrabrykh]

I tried it and for me, it doesn't work. The validator doesn't see the errors, but pipeline doesn't start at all: validation error in the pipeline where I tried to reuse a script.

[Aleksander Parachniewicz]

This does not work, multiline setup-private-key script will be joined into a one line and failed:

    - script: &setup-private-key
                echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
                chmod 600 ~/.ssh/id_rs

Will result to:

 echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa chmod 600 ~/.ssh/id_rs

There is no obvious solution of the problem at the moment, a simplified code:

definitions:
     scripts:
         - script: &script1
                           echo script1
                           echo $MYVAR
         - script: &setup-var
                           export MYVAR=111
    steps:
        - step: &step1
              name: Step1
              script:
                  - *setup-var
                  - *script1
                  - echo DONE

pipelines:
  default:
            - step:
                  <<: *step1

 Results into a failing code:

echo script1 echo 111

If to decide to join the lines using semicolon, then  the output on BB pipeline page would not be readable.

[Kiran Kshatriya]

Adding semi-colons at the end of each line seems to work fine

[Richard Quadling]

So, I'm NOT an expert!!! But we have had this pipeline (heavily edited) for years. I've not found any documentation that explains it in a useful manner, but here it is!

aws-login: &aws-login |-
  STS=($( \
    aws sts assume-role-with-web-identity \
      --role-session-name bitbucket-management-assume-role \
      --role-arn arn:aws:iam::$OIDC_ACCOUNT_ID:role/identity-provider-bitbucket-assume-role \
      --web-identity-token $BITBUCKET_STEP_OIDC_TOKEN \
      --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
      --output text \
  ))

  if [ $? -ne 0 ]; then
    return 1
  fi

  export AWS_ACCESS_KEY_ID=${STS[0]}
  export AWS_SECRET_ACCESS_KEY=${STS[1]}
  export AWS_SESSION_TOKEN=${STS[2]}

  ASSUME_INTO_ACCOUNT=($( \
    aws sts assume-role \
      --role-session-name bitbucket-account-org-role \
      --role-arn arn:aws:iam::${DEVOPS_AWS_ACCOUNT_ID}:role/DeploymentRole \
      --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
      --output text \
  ))

  if [ $? -ne 0 ]; then
    return 1
  fi

  export AWS_ACCESS_KEY_ID=${ASSUME_INTO_ACCOUNT[0]}
  export AWS_SECRET_ACCESS_KEY=${ASSUME_INTO_ACCOUNT[1]}
  export AWS_SESSION_TOKEN=${ASSUME_INTO_ACCOUNT[2]}

So that's at the top of the bitbucket-pipelines.yml file. Not part of anything other than something that exists in the root of the document.

To use it ...

script:
  - *aws-login

And as it is part of a script, you can add as many things before or after as you see fit.

e.g.

definitions:
  steps:
    - step: &build
        caches:
          - docker
        services:
          - docker
        artifacts:
          - test-reports/**
        oidc: true
        script:
          - *aws-login
          - >
            docker build
            ...

pipelines:
  pull-requests:
    '**':
      - step: *build

  default:
    - step: *build

With master having the "deployment" aspect.

This is NOT a perfect setup, but it does work. Why? Not got a clue!

It looks like the root `aws-login` becomes a thing that is injected exactly where you want it to be injected.

[Kiran]

Can we use after-script in definitions as below and leverage *step_report_publish throughout yml

definitions:
  steps:
    - step: &report_generation
        name: Connect Final results
        size: 2x
        script:
          - tests_merge.sh --merge results

  after-script: &step_report_publish
      - tests_publish.sh --dir target/site --merge results

[matteo_migliaccio]

Moved

[Charlie Jonas]

what if you need to share code across multiple single steps

[Daniel Santos]

Hi @Charlie Jonas

That is not possible through bitbucket-pipelines.yml default coding. What you could do instead is to create a script with the command you want to run and call that script in multiple steps.

There is a feature request that might address your request:

• [BCLOUD-17181] Allow users to define tasks / step configuration which can be reused

I hope that helps.

[Jim Fang]

This ticket is closed without any solution for reuse script in multiple steps

[Ekaterina Khrabrykh]

Is it correct that it is still impossible to define a script and reuse it in several steps? 

[Robin Windey]

Like @Diônitas Mendes dos Santos mentioned I think currently this is only possible if you're able to write your script as "oneliner". As soon as the scripts get's more complicated, it will be a mess. For example in GitLab sharing scripts through YAML anchors is possible. Would be nice to have it in Bitbucket, too.