Building a Bitbucket Pipe as a casual coder

[#1]

When Atlassian's partner team contacted Rollbar in early January about participating in the Bitbucket Pipes launch, the team here saw an exciting opportunity to help our mutual customers simplify their continuous delivery process. Bitbucket Pipelines make it easy to set up a continuous deployment pipeline, and Rollbar provides critical real-time feedback about new exceptions which may be due to the latest code changes.

Our development team was fully booked at the time but we didn't want to pass up the opportunity, so I took on building and launching the Rollbar Pipe as a side project. Thanks to the great tools and support provided by the Atlassian team, even a product person who doesn't code much anymore and knew next to nothing about Docker going in (i.e. me) was able to build and launch a pipe.

This post covers the process of developing and releasing our rollbar/rollbar-notify pipe. Hopefully it'll encourage more dev tool makers to build Bitbucket Pipes, even if they aren't developers themselves!

Getting started w/ development

Every Bitbucket pipe is an instance of a Docker container, so developing a pipe requires learning a bit about Docker. Thankfully Docker provides great documentation for getting started and has a pretty intuitive command line interface, so it wasn't too hard to learn all the concepts and commands relatively quickly.

The Rollbar pipe only does one thing: Send a POST request to our deploy API so that developers can correlate new exceptions with recently deployed code changes. To do this all our Docker image needed was to have curl and jq installed, so we chose to use the Alpine image. our DOCKERFILE looks like:

FROM alpine:3.8
RUN apk update && apk add bash curl && apk add bash jq
CMD /bin/bash
COPY pipe /usr/bin/
RUN chmod +x /usr/bin/pipe.sh
ENTRYPOINT ["/usr/bin/pipe.sh"]

To develop the logic of the pipe, I was able to pull a lot of code from the atlassian/opsgenie-send-alertpipe, which was doing something quite similar to what our pipe needed to do. Atlassian also provided a common.sh file full of convenience functions for formatting messages, running the pipe in debug mode, etc. 

Here's the full content of our pipe.sh: 

#!/bin/bash

source "$(dirname "$0")/common.sh"

enable_debug
extra_args=""
if [[ "${DEBUG}" == "true" ]]; then
  extra_args="--verbose"
fi

# mandatory variables
ROLLBAR_ACCESS_TOKEN=${ROLLBAR_ACCESS_TOKEN:?'ROLLBAR_ACCESS_TOKEN environment variable missing.'}
ROLLBAR_ENVIRONMENT=${ROLLBAR_ENVIRONMENT:?'ROLLBAR_ENVIRONMENT environment variable missing'}

#Derived parameters
LOCAL_USERNAME='Bitbucket Pipelines'
COMMENT="https://bitbucket.org/${BITBUCKET_REPO_OWNER}/${BITBUCKET_REPO_SLUG}/addon/pipelines/home#!/results/${BITBUCKET_BUILD_NUMBER}"

output_file="/tmp/pipe-$RANDOM.txt"

run curl -s --request POST \
      --url https://api.rollbar.com/api/1/deploy/ \
      --output $output_file -w "%{http_code}" \
      --form access_token=$ROLLBAR_ACCESS_TOKEN \
      --form environment=$ROLLBAR_ENVIRONMENT \
      --form revision=$BITBUCKET_COMMIT \
      --form local_username="${LOCAL_USERNAME}" \
      --form comment=$COMMENT \

response=$(cat $output_file)
info "HTTP Response: $(echo $response)"

if [[ "${output}" = 2* ]]; then
  rollbar_deploy_id=$(echo $response | jq -r '.data.deploy_id')
  success "Deploy was successfully reported to Rollbar. You can check your deploy here: https://rollbar.com/deploy/$rollbar_deploy_id"
else
  fail "Something failed. Deploy was not reported to Rollbar."
fi

Here's an example of the output generated when rollbar/rollbar-notify runs:

Testing the pipe

To test both locally and in the release pipeline (more on this later), it works really well to use Bats (Bash Automated Testing System). Here's an example of the Bats file for the Rollbar pipe:

#!/usr/bin/env bats

# NOTE: This test assumes that ROLLBAR_ACCESS_TOKEN has `post_server_item` scope.

setup() {
  DOCKER_IMAGE=${DOCKER_IMAGE:="test/rollbar-notify"}

  echo "Building image $DOCKER_IMAGE..."
  docker build -t ${DOCKER_IMAGE} .
}

@test "Deploy is successfully posted" {

    run docker run \
        -e ROLLBAR_ACCESS_TOKEN="${ROLLBAR_ACCESS_TOKEN}" \
        -e ROLLBAR_ENVIRONMENT="test" \
        -e BITBUCKET_COMMIT="${BITBUCKET_COMMIT}" \
        -e BITBUCKET_REPO_OWNER="${BITBUCKET_REPO_OWNER}" \
        -e BITBUCKET_REPO_SLUG="${BITBUCKET_REPO_SLUG}" \
        -e BITBUCKET_BUILD_NUMBER="${BITBUCKET_BUILD_NUMBER}" \
      ${DOCKER_IMAGE}

    echo "Status: $status"
    echo "Output: $output"

    # Checking for status 0 is a sufficient test because this only happens if
    # a deploy was successfully POSTed.
    [ "$status" -eq 0 ]
}

To speed up the development inner loop, it was helpful to create a simple test.sh shell script with hard-coded environment variables to run the Bats test:

export ROLLBAR_ACCESS_TOKEN="4532fb30a98d4b4faf70464a51b60026"
export ROLLBAR_ENVIRONMENT="test"
export BITBUCKET_COMMIT="12345abcde"
export BITBUCKET_REPO_OWNER="rollbar"
export BITBUCKET_REPO_SLUG="rollbar-notify"
export BITBUCKET_BUILD_NUMBER="1"
bats test/test.bats

A successful run of the test locally would result in the following output:

 

The new deploy could be viewed at https://rollbar.com/Rollbar/bitbucket-pipe-test/deploys/?environments=test.

Building & Releasing the pipe

The Rollbar Pipe is of course hosted in Bitbucket and uses a Bitbucket pipeline to automate testing and release. Here's the pipeline configuration we use, which is based on Atlassian's own pipe implementation:

image: atlassian/default-image:2

test: &test
  step:
    name: Test
    script:
    - npm install -g bats
    - bats test/test.bats
    services:
    - docker

push: &push
  step:
    name: Push and Tag
    image: python:3.6.7
    script:
    - pip install semversioner==0.*
    - ./ci-scripts/bump-version.sh
    - ./ci-scripts/docker-release.sh rollbar/$BITBUCKET_REPO_SLUG
    - ./ci-scripts/git-commit.sh
    - ./ci-scripts/git-tag.sh
    services:
    - docker

pipelines:
  default:
  - <<: *test
  branches:
    master:
    - <<: *test
    - <<: *push

Changes made to any branch other than master will trigger a run of the Bats test, while a change to master triggers both the test and the release process.

Semantic versioning

To simplify the process of managing semantic versioning, Raul Gomis of the Atlassian team created semversioner. It's incredibly handy and Rollbar is even considering to use it for our SDK releases.

To release a new version of the pipe, we simply run a command such as the following in the local dev environment:

$ semversioner add-change --type patch --description "Update curl call to check for API failure"

semversioner handles bumping the version and generating a changelog such as our changelog.md.

Pushing to Docker Hub & Tagging releases

The CI scripts included in the repo (also provided by Atlassian) handle pushing a new docker image to your Docker Hub (you can view ours here) and tagging the git repository to reflect the release. The scripts automatically update the pipe.yml metadata file and the readme.md so they reflect the semantic version of the new release:

Summary

With the helpful tools provided by Atlassian as well as open source projects like Docker and Bats, it's extremely easy to build, release, and support a Bitbucket Pipe. If your team is building tools that can integrate into a continuous deployment pipeline, definitely consider building a Bitbucket Pipe!

[Gonchik Tsymzhitov]

Thanks for that!

Looks like nearest future a lot of job title  will be looks like "Senior YAML solution architect" ;) 

[Brett Mathe]

How would you go about exporting the /tmp/pipe-$RANDOM.txt file as an artifact for use in other pipeline steps?  I'm stuck on that part.  I need that output for my other steps.

[Jesse Jesse]

@Brett Mathe I recall speaking w/ the Atlassian team about that and they said you'd need to write the value to a file that will persist across pipeline steps.  I forget the exact details so you're best bet is to contact Atlassian support.

[Brett Mathe]

I ended up figuring it out, you can just save the txt file to the root of the working directory in the Pipe and then in the step just reference it in the artifacts section.  

If you want the file to be shared by other Pipes you have to save the file to the shared pipes directory which is a default environment variable.