Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

Bitbucket pipeline reuse of code

Deleted user July 19, 2019

I have lot's of duplicate code in my pipeline yaml. I tried the 'YAML anchors and references' approach suggested in Q&A https://community.atlassian.com/t5/Bitbucket-questions/How-to-reuse-steps-for-different-branches/qaq-p/875615 but it doesn't work and I get an error when I want to launch my pipeline.

The error is sometling like:

Configuration error

There is an error in your bitbucket-pipelines.yml at [pipelines > custom > build-internal > 0 > step > script > 1]. To be precise: Missing or empty command string. Each item in this list should either be a single command string or a map defining a pipe invocation.

Validate your bitbucket-pipelines.ymlView bitbucket-pipelines.yml

 

Below is a code sample to test:

image: maven

clone:
depth: 500

commonStep: &commonStep
- step:
script:
- echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
custom:
build-internal:
- step:
name: Build maven internal
caches:
- maven
script:
- echo "Lot's of code redundancy"
- <<: *commonStep

deploy-internal-tst:
- step:
name: Deploy to test
deployment: test
script:
- echo "Lot's of code redundancy"
- <<: *commonStep

What is going wrong?

5 answers

1 accepted

3 votes
Answer accepted
Daniel Santos
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 22, 2019

Hi @[deleted]

I think the problem you are facing is caused by indentation. The common step behaves like a full step, not only an extra command in the script section. 

I understand you when you say that there is no validation error in the syntax but there are compilation errors. This is caused by a problem to validate files using anchors. This problem is already reported here:

I've run this test and it worked as expected. Please notice that I used the default section to simplify running it locally.

image: maven

clone:
depth: 500

commonStep: &commonStep
  step:
    script:
      - echo "How come this causes a -Configuration error- when I try to execute a pipeline while it did pass the in-browser YAML validation?"

pipelines:
  default:
    - step:
        name: Build maven internal
        caches:
          - maven
        script:
          - echo "Lot's of code redundancy"
    - <<: *commonStep

You will see that

 - <<: *commonStep

is aligned with the above step and it will create an extra step when compiling.

 

I also built another 2 tests examples to show a possible way to use anchors.

The first one is similar to the one shared in that question you mentioned:

commonStep: &commonStep
  step:
    script:
      - echo "common step"

commonScript: &commonScript
  script:
    - echo "common script"

pipelines:
  default:
    - <<: *commonStep
    - step:
        <<: *commonScript

The second one is similar to what is shared in our docs:

definitions:
  step: &myCommonStep
    script: &myCommonScript
      - echo "definition command"

pipelines:
  default:
    - step: *myCommonStep
    - step:
        script: *myCommonScript
    - step:
        <<: *myCommonStep
        name: my new name

Reference: YAML anchors - Atlassian Documentation

I tested both of them.

I hope that helps.

Deleted user July 23, 2019

Thanks Daniel, your reply got me going.

Rewriting my example the now working version looks like this:

# Docker image
image: maven

clone:
depth: 500

commonStep1: &commonStep1
- step:
name: Step 1a
script:
- echo "Lots of duplicate code"
- echo "Let's see the file structure"
- mkdir -p test
- echo "." > test/test-file2.txt
- ls -la test/
artifacts:
- test/*.txt
commonStep2: &commonStep2
- step:
name: Step 1b
script:
- echo "Even more duplicate code"
- echo "Let's see the file structure"
- mkdir -p test
- ls -la test/
pipelines:
custom:
build-internal:
- step:
name: Step 1
script:
- echo "Let's get started"
- mkdir -p test
- echo "." > test/test-file1.txt
- ls -la test/

- <<: *commonStep1
- <<: *commonStep2
Like # people like this
Daniel Santos
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
July 23, 2019

Hi @[deleted]

I'm glad you were already finding your own way!
I wanted to share those examples as a reference for other users as well.

Thank you for taking the time to create this question and following up on it, I appreciate that.

Have a good one!

Like Denis Baltor likes this
rgbp
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
February 13, 2020

Hi @Daniel Santos ,

I am trying to reuse a common script and add more script commands on the same step, but I am not able to do it. Is it possible? Here's what I am trying to do, from your example:

definitions:
scripts:

  script: &commonScript
    - echo "common script"

pipelines:
  default:
    - step:
script:
        <<: *commonScript
- echo "another script"
Like # people like this
Diônitas Mendes dos Santos
Contributor
June 22, 2020

@rgbp

I was having the same problem, this solves for me. take a try.

definitions:
script: &commonScript
echo "common script"

pipelines:
default:
- step:
script:
- *commonScript
- echo "another script"
Like # people like this
Khalifa EL BANAN October 12, 2020

@Diônitas Mendes dos Santos I tried your solution after a plenty of tries but I get a configuration error at the beginning of the pipeline. Even if BB validator says that everything is fine with my pipeline.

PS: I'm using parallel steps

Like # people like this
raymond Kelly October 12, 2020

 @Khalifa EL BANAN  can you provide an example of your pipeline as I got it working. Note: I did see an error when I tried to make the reusable script a list and instead needed to make them a string and use &&

Like Khalifa EL BANAN likes this
Khalifa EL BANAN October 12, 2020

Yes sure, in general my pipeline looks like the following:

definitions:
script: &myCommonScript
- first script
- second script

pipelines:
pull-requests:
feature/*:
- step:
name: 'first step'
script:
- some script

- parallel:
- step:
name: 'parallel step 1'
script:
- *myCommonScript
- a further script
- step:
name: 'parallel step 2'
script:
- *myCommonScript
- another script
raymond Kelly October 12, 2020

The issue is likely that 

&myCommonScript

is a list. When you try to use that in a script such as

        - step:
name: 'parallel step 1'
script:
- *myCommonScript
- a further script

what you actually have is a list of scripts within a script. My recommendation would instead be changing your pipeline to have the common script be a single operation such as

definitions:
script: &myCommonScript
first script &&
second script

pipelines:
pull-requests:
feature/*:
- step:
name: 'first step'
script:
- some script

- parallel:
- step:
name: 'parallel step 1'
script:
- *myCommonScript
- a further script
- step:
name: 'parallel step 2'
script:
- *myCommonScript
- another script

 There may be a cleaner way of doing this but I have not found it

Like # people like this
Khalifa EL BANAN October 14, 2020

thanks @raymond Kelly 

Like raymond Kelly likes this
Jim Fang April 9, 2021

It doesn't work. 

 

Configuration error

There is an error in your bitbucket-pipelines.yml at [pipelines > branches > TEST-DEV > 0 > step > script > 0]. To be precise: This section should be a string or a map (it is currently defined as a list).

Diônitas Mendes dos Santos
Contributor
April 9, 2021

Hi Jim, note in my case a not using a list, is just a string. if you need more than one script in commonScript, you can do some thing like that

definitions:
script: &commonScript
echo "common script 1";
echo "common script 2";


pipelines:
default:
- step:
script:
- *commonScript
- echo "another script"

the yaml parse will treat as if it were a big string



echo "common script 1";echo "common script 2";
Like # people like this
Jim Fang April 9, 2021

Thanks @Diônitas Mendes dos Santos

i will post my bitbucket-pipelines.yml.  I check with bitbucket validatator. it said "valid", but i tried it with bitbucket pipeline. It reported error.

--------------------

There is an error in your bitbucket-pipelines.yml at [pipelines > branches > TEST-DEV > 0 > step > script > 0]. To be precise: This section should be a string or a map (it is currently defined as a list).

-------------------

```
image : nikolaik/python-nodejs:python3.8-nodejs15

definitions:
step: &myCommonStep
script: &myCommonScript
- curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
- unzip awscliv2.zip
- ./aws/install
- ls -lrta
- npm install -g serverless
- python --version
- serverless --version
- aws --version
- export roleARN=$ROLE_ARN_DEV
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID --profile loop-role
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY --profile loop-role
- echo "role_arn=$roleARN" >> ~/.aws/credentials
- echo "source_profile=default" >> ~/.aws/credentials
- cat ~/.aws/credentials
- aws configure set source_profile default --profile loop-role
- aws configure set role_arn $roleARN --profile loop-role
- aws configure set region ap-southeast-2 --profile loop-role
- aws configure set output json --profile loop-role
- aws configure list
- cat ~/.aws/credentials
- cat ~/.aws/config

pipelines:
branches:
TEST-DEV:
- step:
name: serverless build
caches:
- node
script:
- *myCommonScript
- npm install -g serverless-pseudo-parameters
- npm install -g serverless-python-requirements
- serverless deploy --stage dev --verbose --aws-profile loop-role

```

Jim Fang April 9, 2021
# I tried to use code block, but after copy-paste, it disappear.
# This pipeline is to use serverless and assume role to deploy an application

Thanks 

2 votes
Vytenis Ščiukas July 27, 2021

For anyone that wants undocumented syntax for multiple scripts here is an example:

image: somerunnerimage:latest

definitions:
scripts:
- script: &setup-private-key
echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
- script: &setup-environment
export ENVIRONMENT=${BITBUCKET_DEPLOYMENT_ENVIRONMENT}
steps:
- step: &testing
name: Testing
runs-on:
- "self.hosted"
caches:
- node
- composer
script:
- phing -Denvironment=test
- php artisan test --testsuite=Unit

- step: &deployment
name: Building & deploying
runs-on:
- "self.hosted"
caches:
- node
- composer
script:
- *setup-private-key
- *
setup-environment
- phing -Denvironment=${ENVIRONMENT}
- dep deploy --revision=${BITBUCKET_TAG}

pipelines:
tags:
r[0-9].[0-9].[0-9]:
- step:
<<: *testing
- step:
<<: *deployment
deployment: production
branches
master:
- step:
<<: *deployment
deployment: staging
matteo_migliaccio
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
September 30, 2021

I'm sure we're not alone wondering why this is still undocumented, where can we find more about this and, more importantly, if there's a log somewhere we can monitor to be up to date with new bb pipelines features, we've stumble on this comment just by chance.

Ekaterina Khrabrykh January 13, 2022

I tried it and for me, it doesn't work. The validator doesn't see the errors, but pipeline doesn't start at all: validation error in the pipeline where I tried to reuse a script.

Aleksander Parachniewicz
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
July 9, 2023

This does not work, multiline setup-private-key script will be joined into a one line and failed:

    - script: &setup-private-key
echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rs

Will result to:

 echo ${BITBUCKET_REPO_PRIVATE_KEY} | base64 --decode > ~/.ssh/id_rsa chmod 600 ~/.ssh/id_rs

There is no obvious solution of the problem at the moment, a simplified code:

definitions:
     scripts:
         - script: &script1
                           echo script1
                           echo $MYVAR
         - script: &setup-var
                           export MYVAR=111
    steps:
        - step: &step1
              name: Step1
              script:
                  - *setup-var
                  - *script1
                  - echo DONE

pipelines:
  default:
            - step:
                  <<: *step1

 Results into a failing code:

echo script1 echo 111

If to decide to join the lines using semicolon, then  the output on BB pipeline page would not be readable.

Kiran Kshatriya September 15, 2023

Adding semi-colons at the end of each line seems to work fine

0 votes
Richard Quadling
Contributor
February 1, 2024

So, I'm NOT an expert!!! But we have had this pipeline (heavily edited) for years. I've not found any documentation that explains it in a useful manner, but here it is!

aws-login: &aws-login |-
STS=($( \
aws sts assume-role-with-web-identity \
--role-session-name bitbucket-management-assume-role \
--role-arn arn:aws:iam::$OIDC_ACCOUNT_ID:role/identity-provider-bitbucket-assume-role \
--web-identity-token $BITBUCKET_STEP_OIDC_TOKEN \
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
--output text \
))

if [ $? -ne 0 ]; then
return 1
fi

export AWS_ACCESS_KEY_ID=${STS[0]}
export AWS_SECRET_ACCESS_KEY=${STS[1]}
export AWS_SESSION_TOKEN=${STS[2]}

ASSUME_INTO_ACCOUNT=($( \
aws sts assume-role \
--role-session-name bitbucket-account-org-role \
--role-arn arn:aws:iam::${DEVOPS_AWS_ACCOUNT_ID}:role/DeploymentRole \
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
--output text \
))

if [ $? -ne 0 ]; then
return 1
fi

export AWS_ACCESS_KEY_ID=${ASSUME_INTO_ACCOUNT[0]}
export AWS_SECRET_ACCESS_KEY=${ASSUME_INTO_ACCOUNT[1]}
export AWS_SESSION_TOKEN=${ASSUME_INTO_ACCOUNT[2]}

So that's at the top of the bitbucket-pipelines.yml file. Not part of anything other than something that exists in the root of the document.

To use it ...

script:
- *aws-login

And as it is part of a script, you can add as many things before or after as you see fit.

e.g.

definitions:
steps:
- step: &build
caches:
- docker
services:
- docker
artifacts:
- test-reports/**
oidc: true
script:
- *aws-login
- >
docker build
...
pipelines:
pull-requests:
'**':
- step: *build

default:
- step: *build

With master having the "deployment" aspect.

This is NOT a perfect setup, but it does work. Why? Not got a clue!

It looks like the root `aws-login` becomes a thing that is injected exactly where you want it to be injected.

0 votes
Kiran August 19, 2020

Can we use after-script in definitions as below and leverage *step_report_publish throughout yml

definitions:
steps:
- step: &report_generation
name: Connect Final results
size: 2x
script:
- tests_merge.sh --merge results

after-script: &step_report_publish
- tests_publish.sh --dir target/site --merge results
matteo_migliaccio
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
September 30, 2021

Moved

 

0 votes
Charlie Jonas November 13, 2019

what if you need to share code across multiple single steps

Daniel Santos
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
November 20, 2019

Hi @Charlie Jonas

That is not possible through bitbucket-pipelines.yml default coding. What you could do instead is to create a script with the command you want to run and call that script in multiple steps.

There is a feature request that might address your request:

I hope that helps.

Jim Fang April 9, 2021

This ticket is closed without any solution for reuse script in multiple steps

Ekaterina Khrabrykh January 13, 2022

Is it correct that it is still impossible to define a script and reuse it in several steps? 

Robin Windey
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
October 19, 2022

Like @Diônitas Mendes dos Santos mentioned I think currently this is only possible if you're able to write your script as "oneliner". As soon as the scripts get's more complicated, it will be a mess. For example in GitLab sharing scripts through YAML anchors is possible. Would be nice to have it in Bitbucket, too.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events