nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Bitbucket pipeline reuse of code

unknown

I have lot's of duplicate code in my pipeline yaml. I tried the 'YAML anchors and references' approach suggested in Q&A https://community.atlassian.com/t5/Bitbucket-questions/How-to-reuse-steps-for-different-branches/qaq-p/875615 but it doesn't work and I get an error when I want to launch my pipeline.

The error is sometling like:

Configuration error

There is an error in your bitbucket-pipelines.yml at [pipelines > custom > build-internal > 0 > step > script > 1]. To be precise: Missing or empty command string. Each item in this list should either be a single command string or a map defining a pipe invocation.

Validate your bitbucket-pipelines.yml View bitbucket-pipelines.yml

Below is a code sample to test:

image: maven

clone:
 depth: 500

commonStep: &commonStep 
 - step:
  script:
   - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
 custom:
  build-internal:
   - step:
    name: Build maven internal
    caches:
     - maven
    script:
     - echo "Lot's of code redundancy"
     - <<: *commonStep

  deploy-internal-tst:
   - step:
    name: Deploy to test
    deployment: test
    script:
     - echo "Lot's of code redundancy"
     - <<: *commonStep

What is going wrong?

Accepted answers

Daniel Santos_157bm4

Hi @[deleted]

I think the problem you are facing is caused by indentation. The common step behaves like a full step, not only an extra command in the script section.

I understand you when you say that there is no validation error in the syntax but there are compilation errors. This is caused by a problem to validate files using anchors. This problem is already reported here:

Better YML validation when configuration uses YML anchors and references

I've run this test and it worked as expected. Please notice that I used the default section to simplify running it locally.

image: maven

clone:
  depth: 500

commonStep: &commonStep
  step:
    script:
      - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
  default:
    - step:
        name: Build maven internal
        caches:
          - maven
        script:
          - echo "Lot's of code redundancy"
    - <<: *commonStep

You will see that

 - <<: *commonStep

is aligned with the above step and it will create an extra step when compiling.

I also built another 2 tests examples to show a possible way to use anchors.

The first one is similar to the one shared in that question you mentioned:

commonStep: &commonStep
  step:
    script:
      - echo "common step"

commonScript: &commonScript
  script:
    - echo "common script"

pipelines:
  default:
    - <<: *commonStep
    - step:
        <<: *commonScript

The second one is similar to what is shared in our docs:

definitions:
  step: &myCommonStep
    script: &myCommonScript
      - echo "definition command"

pipelines:
  default:
    - step: *myCommonStep
    - step:
        script: *myCommonScript
    - step:
        <<: *myCommonStep
        name: my new name

Reference: YAML anchors - Atlassian Documentation

I tested both of them.

I hope that helps.

All comments

Daniel Santos_157bm4

Hi @[deleted]

Better YML validation when configuration uses YML anchors and references

I've run this test and it worked as expected. Please notice that I used the default section to simplify running it locally.

image: maven

clone:
  depth: 500

commonStep: &commonStep
  step:
    script:
      - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
  default:
    - step:
        name: Build maven internal
        caches:
          - maven
        script:
          - echo "Lot's of code redundancy"
    - <<: *commonStep

You will see that

 - <<: *commonStep

is aligned with the above step and it will create an extra step when compiling.

I also built another 2 tests examples to show a possible way to use anchors.

The first one is similar to the one shared in that question you mentioned:

commonStep: &commonStep
  step:
    script:
      - echo "common step"

commonScript: &commonScript
  script:
    - echo "common script"

pipelines:
  default:
    - <<: *commonStep
    - step:
        <<: *commonScript

The second one is similar to what is shared in our docs:

definitions:
  step: &myCommonStep
    script: &myCommonScript
      - echo "definition command"

pipelines:
  default:
    - step: *myCommonStep
    - step:
        script: *myCommonScript
    - step:
        <<: *myCommonStep
        name: my new name

Reference: YAML anchors - Atlassian Documentation

I tested both of them.

I hope that helps.

unknown

Thanks Daniel, your reply got me going.

Rewriting my example the now working version looks like this:

# Docker image
image: maven

clone:
 depth: 500

commonStep1: &commonStep1
 - step:
  name: Step 1a
  script:
   - echo "Lots of duplicate code"
   - echo "Let's see the file structure"
   - mkdir -p test
   - echo "." > test/test-file2.txt
   - ls -la test/
  artifacts:
   - test/*.txt
commonStep2: &commonStep2
 - step:
  name: Step 1b
  script:
   - echo "Even more duplicate code"
   - echo "Let's see the file structure"
   - mkdir -p test
   - ls -la test/
pipelines:
 custom:
  build-internal:
   - step:
    name: Step 1
    script:
     - echo "Let's get started"
     - mkdir -p test
     - echo "." > test/test-file1.txt
     - ls -la test/

   - <<: *commonStep1
   - <<: *commonStep2

Daniel Santos_157bm4

Hi @[deleted]

I'm glad you were already finding your own way!
I wanted to share those examples as a reference for other users as well.

Thank you for taking the time to create this question and following up on it, I appreciate that.

Have a good one!

Charlie Jonas

what if you need to share code across multiple single steps

Daniel Santos_157bm4

Hi @Charlie Jonas

That is not possible through bitbucket-pipelines.yml default coding. What you could do instead is to create a script with the command you want to run and call that script in multiple steps.

There is a feature request that might address your request:

[BCLOUD-17181] Allow users to define tasks / step configuration which can be reused

I hope that helps.

rgbp

Hi @Daniel Santos ,

I am trying to reuse a common script and add more script commands on the same step, but I am not able to do it. Is it possible? Here's what I am trying to do, from your example:

definitions:
  scripts: 
    script: &commonScript
      - echo "common script"

pipelines:
  default:
    - step:
        script:
          <<: *commonScript
          - echo "another script"

Kiran_qqg0f2

Can we use after-script in definitions as below and leverage *step_report_publish throughout yml

definitions:
steps:
    - step: &report_generation
        name: Connect Final results
size: 2x
script:
          - tests_merge.sh --merge results

after-script: &step_report_publish
      - tests_publish.sh --dir target/site --merge results

Jim Fang_de3c7r

This ticket is closed without any solution for reuse script in multiple steps

Vytenis Ščiukas_36my5b

For anyone that wants undocumented syntax for multiple scripts here is an example:

image: somerunnerimage:latest

definitions:
scripts:
    - script: &setup-private-key
                echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
                chmod 600 ~/.ssh/id_rsa
    - script: &setup-environment
                export ENVIRONMENT=${BITBUCKET_DEPLOYMENT_ENVIRONMENT}
steps:
    - step: &testing
        name: Testing
runs-on:
          - "self.hosted"
        caches:
          - node
          - composer
        script:
          - phing -Denvironment=test
      - php artisan test --testsuite=Unit
  
    - step: &deployment
        name: Building & deploying
runs-on:
          - "self.hosted"
        caches:
          - node
          - composer
script:
- *setup-private-key
          - *setup-environment
          - phing -Denvironment=${ENVIRONMENT}
          - dep deploy --revision=${BITBUCKET_TAG}

pipelines:
tags:
r[0-9].[0-9].[0-9]:
      - step:
<<: *testing
      - step:
<<: *deployment
          deployment: production
branches:    
master:
      - step:
<<: *deployment
          deployment: staging

matteo_migliaccio

Moved

matteo_migliaccio

I'm sure we're not alone wondering why this is still undocumented, where can we find more about this and, more importantly, if there's a log somewhere we can monitor to be up to date with new bb pipelines features, we've stumble on this comment just by chance.

Ekaterina Khrabrykh

Is it correct that it is still impossible to define a script and reuse it in several steps?

Ekaterina Khrabrykh

I tried it and for me, it doesn't work. The validator doesn't see the errors, but pipeline doesn't start at all: validation error in the pipeline where I tried to reuse a script.

Aleksander Parachniewicz

This does not work, multiline setup-private-key script will be joined into a one line and failed:

    - script: &setup-private-key
                echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
                chmod 600 ~/.ssh/id_rs

Will result to:

 echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa chmod 600 ~/.ssh/id_rs

There is no obvious solution of the problem at the moment, a simplified code:

definitions:
     scripts:
         - script: &script1
                           echo script1
                           echo $MYVAR
         - script: &setup-var
                           export MYVAR=111
    steps:
        - step: &step1
              name: Step1
              script:
                  - *setup-var
                  - *script1
                  - echo DONE

pipelines:
  default:
            - step:
                  <<: *step1

Results into a failing code:

echo script1 echo 111

If to decide to join the lines using semicolon, then the output on BB pipeline page would not be readable.

Richard Quadling_ut6oj3

So, I'm NOT an expert!!! But we have had this pipeline (heavily edited) for years. I've not found any documentation that explains it in a useful manner, but here it is!

aws-login: &aws-login |-
  STS=($( \
    aws sts assume-role-with-web-identity \
      --role-session-name bitbucket-management-assume-role \
      --role-arn arn:aws:iam::$OIDC_ACCOUNT_ID:role/identity-provider-bitbucket-assume-role \
      --web-identity-token $BITBUCKET_STEP_OIDC_TOKEN \
      --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
      --output text \
  ))

  if [ $? -ne 0 ]; then
    return 1
  fi

  export AWS_ACCESS_KEY_ID=${STS[0]}
  export AWS_SECRET_ACCESS_KEY=${STS[1]}
  export AWS_SESSION_TOKEN=${STS[2]}

  ASSUME_INTO_ACCOUNT=($( \
    aws sts assume-role \
      --role-session-name bitbucket-account-org-role \
      --role-arn arn:aws:iam::${DEVOPS_AWS_ACCOUNT_ID}:role/DeploymentRole \
      --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
      --output text \
  ))

  if [ $? -ne 0 ]; then
    return 1
  fi

  export AWS_ACCESS_KEY_ID=${ASSUME_INTO_ACCOUNT[0]}
  export AWS_SECRET_ACCESS_KEY=${ASSUME_INTO_ACCOUNT[1]}
  export AWS_SESSION_TOKEN=${ASSUME_INTO_ACCOUNT[2]}

So that's at the top of the bitbucket-pipelines.yml file. Not part of anything other than something that exists in the root of the document.

To use it ...

script:
  - *aws-login

And as it is part of a script, you can add as many things before or after as you see fit.

e.g.

definitions:
steps:
    - step: &build
        caches:
          - docker
services:
          - docker
artifacts:
          - test-reports/**
oidc: true
script:
          - *aws-login
          - >
docker build
            ...

pipelines:
pull-requests:
'**':
      - step: *build

  default:
    - step: *build

With master having the "deployment" aspect.

This is NOT a perfect setup, but it does work. Why? Not got a clue!

It looks like the root `aws-login` becomes a thing that is injected exactly where you want it to be injected.

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket

bitbucket-pipelines

atl-pipelines

duplicate

anchor

yaml

reference