Waiting for dependent services, like databases

[Daniel_Bunte]

Hi there,

I'm pretty new to Bitbucket pipelines, but have worked extensively with Jenkins Pipelines before.

For my current project (Angular4, Spring-Boot, MariaDB), I have set up a pipeline with a custom Docker container, linked to MariaDB via linking (database service in pipelines).

I encountered problems with randomly failing builds, because the MariaDB container wasn't ready at the time the actual maven build started. Therefore, I was looking to find a suitable solution and wrote a script that I'd like to share. I hope it comes in handy for you :)

File: wait-for-mysql.sh

#!/bin/bash -e

statusFile=/tmp/mysql-status
while [[ true ]]; do
  telnet 127.0.0.1 3306 &> ${statusFile}
  status=$(grep "Connection refused" ${statusFile} | wc -l)
  echo "Status: $status"

  if [[ "${status}" -eq 1 ]]; then
    echo "MySQL not running, waiting."
    sleep 1
  else
    rm ${statusFile}
    echo "MySQL running, ready to proceed."
    break;
  fi
done

 File: bitbucket-pipelines.yml

image: alphabrik/maven-angular-bootstrap:latest

pipelines:
  default:
    - step:
        caches:
          - maven
          - node
          - node-app
        script:
          - bash wait-for-mysql.sh
          - mvn install
        services:
          - database

definitions:
  caches:
    node-app: my-app/node_modules
  services:
    database:
      image: mariadb:10.1
      environment:
        MYSQL_DATABASE: 'awesomeDBName'
        MYSQL_USER: 'youruser'
        MYSQL_PASSWORD: 'yourpassword'
        MYSQL_ALLOW_EMPTY_PASSWORD: 'true'
        volumes: path/to/your/init-dump.sql:/docker-entrypoint-initdb.d

 Now, how does it work?

My custom docker image includes everyhting needed for testing Spring and Angular apps. If you're using some different image, make sure it contains the 'telnet' program.

The pipeline steps will call the above mentioned script, before maven is executed. The script attempts a connection to the mysql-server via telnet in a loop. Once the connection is successful, the build can continue.

Since bitbucket pipelines currently doesn't offer a method to wait for services, I thought I'd start this discussion and see what kind of solutions the community can find for other dependent services.

Feel free to ask any questions!

Cheers,

Daniel

[Deleted user]

Hey @Mario Staykov, shouldn't be "${status} -ne 0" ?

[Mario Staykov]

Hi all. Thanks for the script Daniel, that was almost exactly what I needed. I found that despite the "&" backgrounding, telnet would keep the script running even when succeeding in its connection, so I amended the telnet line to exit immediately upon success.

Additionally, through some trial and error I confirmed my suspicion that the approach of calling it a success when "Connection refused" doesn't appear is flawed, since it passes at system startup - probably due to a message like "No route to host". As a result, I have amended your script to the following:

#!/bin/bash -e

statusFile=/tmp/mysql-status
while [[ true ]]; do
 echo "" | telnet db2.novarumcloud.com 5432 &> ${statusFile}
 status=$(grep "Connected to" ${statusFile} | wc -l)
 echo "Status: $status"

 if [[ "${status}" -ne 1 ]]; then
   echo "MySQL not running, waiting."
   sleep 1
 else
   rm ${statusFile}
   break;
 fi
done

Apart from that, I'd like to mention where I had to insert this holder script on my local JIRA installation, in case anyone else needs to figure out the same:

/etc/init.d/jira1

# JIRA Linux service controller script
cd "/opt/atlassian/jira-core/bin"

case "$1" in
 start)
   bash wait-for-mysql.sh
   ./start-jira.sh
   ;;
 stop)
   ./stop-jira.sh
   ;;
 *)
   echo "Usage: $0 {start|stop}"
   exit 1
   ;;
esac

Hope that expands the value of an already useful answer to a wider audience.

[Stephen Fox]

Yes, it would be nice if Pipelines could have something like a serviceready feature instead of having to write custom scripts to poll if the service has started.

services:
    database:
      image: mariadb:10.1
      environment:
        ...
      serviceready:
        script:
          sh test_my_sql.sh