Improve the pipeline

[Deleted user]

Hello,

So I'm creating CICD with bitbucket pipelines and digital ocean.

my step are to create an image and push it.

to do this, I need to install `doctl` and login.

is there any way that I can keep doctl cached to be used with other steps instead of each time installing it?

Also, if possible to know, in the below PR section, is it running in case of PRs on the development branch as well? although I'm protecting the development to accept only PRs?

if so, is there a way to run all PR except for `master` and `dev`?

here is my yml

#  Template NodeJS build

#  This template allows you to validate your NodeJS code.
#  The workflow allows running tests and code linting on the default branch.

image: node:16

pipelines:
  pull-requests:
    '**':
      - step:
          name: Build, Lint and test
          caches:
            - node
          script:
            - npm install
            - npm run precommit
            - npm test

  branches:
    dev:
      - step:
          name: Build, Lint and test
          caches:
            - node
          script:
            - npm install
            - npm run precommit
            - npm test
      - step:
          name: Build docker and Test
          script:
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG
            - docker build . --file ./docker/dev/backend/Dockerfile --tag ${IMAGE_NAME}
            - docker save ${IMAGE_NAME} --output "${IMAGE_NAME}.tar"
          services:
            - docker
          caches:
            - docker
          artifacts:
            - "*.tar"
      - step:
          name: push image
          services:
            - docker
          script:
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG
            - VERSION="dev-0.1.${BITBUCKET_BUILD_NUMBER}"
            - IMAGE=${DOCKERHUB_NAMESPACE}/${IMAGE_NAME}
            - wget https://github.com/digitalocean/doctl/releases/download/v1.86.0/doctl-1.86.0-linux-amd64.tar.gz
            - tar xf doctl-1.86.0-linux-amd64.tar.gz
            - mv doctl /usr/local/bin
            - which doctl
            - docker load --input "${IMAGE_NAME}.tar"
            - doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN
            - doctl account get
            - doctl registry login --expiry-seconds 600
            - docker tag "${IMAGE_NAME}" "${IMAGE}:${VERSION}"
            - docker push "${IMAGE}:${VERSION}"
      - step:
          name: deploy to development
          deployment: Staging
          services:
            - docker
          script:
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG
            - IMAGE=${DOCKERHUB_NAMESPACE}/${IMAGE_NAME}
            - VERSION="dev-0.1.${BITBUCKET_BUILD_NUMBER}"
            - pipe: atlassian/ssh-run:0.4.1
              variables:
                SSH_USER: 'root'
                SERVER: $HOST
                COMMAND: '
                doctl registry login --expiry-seconds 600;
                docker rm -f app_dev  2> /dev/null;
                docker run  --name app_dev -d "${IMAGE}:${VERSION}"
                docker network connect app_dev-network app_dev
                '
                SSH_KEY: $SSH_PRIVATE_KEY
                EXTRA_ARGS: '-o StrictHostKeyChecking=no'
                ENV_VARS: >-
                  IMAGE=${IMAGE} 
                  VERSION=${VERSION}
                  DB_HOST=${DB_HOST}
                  DB_PORT=${DB_PORT}
                  DB_USER=${DB_USER}
                  DB_PASS=${DB_PASS}
                  DB_NAME=${DB_NAME}
                  APP_PORT=${APP_PORT}
    master:
      - step:
          name: Build, Lint and test
          caches:
            - node
          script:
            - npm install
            - npm run precommit
            - npm test
      - step:
          name: Build docker and Test
          script:
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG
            - wget https://github.com/digitalocean/doctl/releases/download/v1.86.0/doctl-1.86.0-linux-amd64.tar.gz
            - tar xf doctl-1.86.0-linux-amd64.tar.gz
            - mv doctl /usr/local/bin
            - which doctl
            - doctl registry login
            - docker build . --file ./docker/prod/backend/Dockerfile --tag ${IMAGE_NAME}
            - docker save ${IMAGE_NAME} --output "${IMAGE_NAME}.tar"
          services:
            - docker
          caches:
            - docker
          artifacts:
            - "*.tar"
      - step:
          name: push image
          services:
            - docker
          script:
            - IMAGE_NAME=$BITBUCKET_REPO_SLUG
            - VERSION="prod-0.1.${BITBUCKET_BUILD_NUMBER}"
            - IMAGE=${DOCKERHUB_NAMESPACE}/${IMAGE_NAME}
            - wget https://github.com/digitalocean/doctl/releases/download/v1.86.0/doctl-1.86.0-linux-amd64.tar.gz
            - tar xf doctl-1.86.0-linux-amd64.tar.gz
            - mv doctl /usr/local/bin
            - which doctl
            - docker load --input "${IMAGE_NAME}.tar"
            - doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN
            - doctl registry login --expiry-seconds 600
            - docker tag "${IMAGE_NAME}" "${IMAGE}:${VERSION}"
            - docker push "${IMAGE}:${VERSION}"
      - step:
          name: deploy to production
          trigger: manual
          deployment: Production
          services:
            - docker
          script:
            - IMAGE=${DOCKERHUB_NAMESPACE}/${IMAGE_NAME}
            - VERSION="prod-0.1.${BITBUCKET_BUILD_NUMBER}"
            - pipe: atlassian/ssh-run:0.4.1
              variables:
                SSH_USER: 'root'
                SERVER: $HOST
                COMMAND: '
                 doctl registry login --expiry-seconds 600;
                 docker rm -f app_prod  2> /dev/null;
                 docker run --name app_prod -d "${IMAGE}:${VERSION}"
                 docker network connect app_dev-network app_prod
                 '
                SSH_KEY: $SSH_PRIVATE_KEY
                EXTRA_ARGS: '-o StrictHostKeyChecking=no'
                ENV_VARS: >-
                  IMAGE=${IMAGE} 
                  VERSION=${VERSION}
                  DB_HOST=${DB_HOST}
                  DB_PORT=${DB_PORT}
                  DB_USER=${DB_USER}
                  DB_PASS=${DB_PASS}
                  DB_NAME=${DB_NAME}
                  APP_PORT=${APP_PORT}

[Patrik S · Accepted solution]

Hello @[deleted] ,

Welcome to Atlassian Community!

In order to not have to download and install the doctl every step, you have the option to build your custom docker image with doctl installed on it, and use this custom docker image as part of your build.

Currently you are using node:16 as your build image, so allow me to share an example of how you could build a custom docker image : 

1. Create a file named Dockerfile in your local : 

touch Dockerfile

 2. Edit the file with the following content :

FROM node:16

RUN wget https://github.com/digitalocean/doctl/releases/download/v1.86.0/doctl-1.86.0-linux-amd64.tar.gz

RUN tar xf doctl-1.86.0-linux-amd64.tar.gz

RUN mv doctl /usr/local/bin


This will base the custom image in node:16 and install the doctl on it.

3. Save the file and execute the following command in your terminal in the same folder where Dockerfile is : 

docker build -t <docker_hub_namespace>/<custom docker image name>

4. Push the image to Docker Hub

docker push <docker_hub_namespace>/<custom docker image name>

5. Use the image in your pipeline YML file.

image: <custom docker image name>

As for your pull request triggered pipeline questions, the way you currently set up with the pattern '**' will make this pipeline run for pull requests in any branches of your repository.

If you want to configure the pull request pipeline to run just with a subset of branches, you can use a glob pattern like the below example : 

pipelines:
  pull-requests:
    'feature/*': # any branch with a feature prefix
      - step:
          name: Build, Lint and test
          caches:
            - node
          script:
            - npm install
            - npm run precommit
            - npm test
     'bugfix/*,release/*,build/*}': # any branch with bugfix or release or build prefix
      - step:
           name: Build, Lint and test
           script:
             - echo "my pull request build"

For more details about glob patterns, please refer to Use glob patterns on pipeline YML file 

Hope that helps!

Thank you @[deleted] .

Patrik S

[Deleted user]

Thanks @Patrik S for the detailed response. 

Would be really great if there was a pattern to exclude branches rather than including only.