What is the difference in syntax between bitbucket-pipelines services and docker-compose?

Antoine Drabble

I'm trying to setup a minIO service for the test pipeline in my project on Bitbucket. I have almost exactly the same configuration in my docker-compose.yml. It works great with Docker-compose.

When I run this in a bitbucket-pipelines.yml, it fails, showing me the usage of the mc command.

My question is : what is the difference in syntax between bitbucket-pipelines and docker-compose?

I have tried using command instead of entrypoint, without success.

definitions:
  services:
    minio:
      image: minio/minio:edge
      container_name: minio
      memory: 1024 # default value
      expose:
        - "9000"
      environment:
        MINIO_ACCESS_KEY: minio
        MINIO_SECRET_KEY: minio123
      command: server /data
    mc:
      image: minio/mc:latest
      memory: 512
      depends_on:
        - minio
      entrypoint: >
        /bin/sh -c "/usr/bin/mc config host rm local
                    /usr/bin/mc config host add --quiet --api s3v4 local http://minio:9000 minio minio123
                    /usr/bin/mc rb --force local/ciip-support/
                    /usr/bin/mc mb --quiet local/ciip-support/"

 

 

ktomk

Hey @Antoine Drabble, welcome to Atlassian community

My question is : what is the difference in syntax between bitbucket-pipelines and docker-compose?

There are no syntax differences on the level of YAML, both Docker Compose and Bitbucket Pipelines are using the same YAML file-format.

However they are entirely different on the level of the Docker Compose and Bitbucket Pipelines own syntax and both have very different semantics. Just copying over from one file to another won't do it as you already have found out.

If I may take the YAML from your question as an example, Docker Compose container representation is services where in Bitbucket pipelines it is foremost pipelines, the meaning of services in definitions in Bitbucket Pipelines may appear similar but is different (compare the docs of Docker Compose and Bitbucket Pipelines for the full picture of the differences).

For example: In Bitbucket Pipelines you can create a pipeline that is running docker compose. Some references about this:

• Run Docker containers and docker-compose files (Dec 2017 by Matt Ryall for Bitbucket)
• Run Docker commands in Bitbucket Pipelines (Atlassian Bitbucket Cloud Support docs)

Alternative to that is having a pipeline with the build image (minio/mc:latest in your case?) making use of a service defined from the service image (minio/minio:edge).

What is the entrypoint might eventually become the script of the pipeline. 

Example 1: Minio Service in Bitbucket Pipelines

Porting your Docker Compose example to Bitbucket Pipelines as a quick command-line example (making use of the pipelines utililty and the shell, it includes the Bitbucket Pipelines yaml as a here document):

$ <<BBPL pipelines --file - 
---
# Example pipelines file with minio service
pipelines:
  default:
    - step:
      name: minio service example
      image: minio/mc:latest
      script:
        - mc config host rm local
        - mc config host add --api s3v4 local http://127.0.0.1:9000 minio minio123
        - mc rb --force local/ciip-support/ || true
        - mc mb local/ciip-support/
      services:
        - minio
definitions:
  services:
    minio:
      image: ktomk/pipelines:minio-server # from minio/minio:edge
      variables:
        MINIO_ACCESS_KEY: minio
        MINIO_SECRET_KEY: minio123
BBPL

 +++ step #1

name...........: "minio service example"
effective-image: minio/mc:latest
container......: pipelines-1.minio-service-example.default.pipelines
container-id...: fb424c4dcfda

 +++ copying files into container...


 + mc config host rm local
Removed `local` successfully.

 + mc config host add --api s3v4 local http://127.0.0.1:9000 minio minio123
Added `local` successfully.

 + mc rb --force local/ciip-support/ || true
mc: <ERROR> Unable to validate target `local/ciip-support/`. Bucket `ciip-support` does not exist.

 + mc mb local/ciip-support/
Bucket created successfully `local/ciip-support/`.
➜ pipelines git:(test) ✗ 

Example Notes:

• First code-block is the command, `$` symbolizes the shell prompt
• Second code-block is the output of that command
• The image `docker.io/minio/minio:edge` can not be used as-is for a Bitbucket Pipelines service as far I could find out quickly. The reason is the `command:` part in the Docker Compose file, the `CMD ["minio", "server", "/data"]` can be moved into a dedicated service image (here  ktomk/pipelines:minio-server, I'll add two-line dockerfile below)
• Technically running both `mc config host rm local` and `mc rb --force local/ciip-support/` is not necessary in Bitbucket Pipelines as the Containers are always starting fresh. This is the reason why the bucket removing command is suffixed with `|| true` as the command to remove the bucket ("rb") does always fail and would otherwise stop the pipeline script. (Nevertheless it's fine to have these when keeping containers with the pipelines utility which has an option for that and it speeds up writing the example, so I have kept it in)

This example shows how it could look like in Bitbucket Pipelines compared to the Docker Compose example/port I could gather from the code-block in your question.

As noted, the Minio service docker container image needs to be tweaked a bit to work out of the box with Bitbucket Pipelines. Here a Dockerfile:

FROM docker.io/minio/minio:edge
CMD ["minio", "server", "/data"]

Reason is that Bitbucket Pipelines for the service definitions has no counter-part to the `command` property in Docker Compose services. It is much more simplified and expects service container images to be configurable with environment variables (this is normally common, I personally do not know Minio too much in this regard so writing the two-line dockerfile was just easier than searching the Minio environment parameters for the "server" subcommand and to specifiy the storage location for this example).

So just take it as what it is, an example that can be validated, for demonstration purposes.

The image `ktomk/pipelines:minio-server` is available on Docker Hub so it can be tested on Bitcuket Pipelines as well, but I have skipped this step so far while writing the answer (edit: it runs flawlessly on Bitbucket Pipelines as well, verfieid it after the initial posting).

Example 2: Minio Docker Compose inside Bitbucket Pipelines

This is the variant mentioned earlier to run `docker compose` in a pipeline. Again a quick command-line example with the pipelines utility. Note that this time you need to have the `docker` service in the pipeline step:

$ <<BBPL pipelines --file -
---
# Example pipelines file with docker compose
pipelines:
  default:
    - step:
      name: docker compose example
      image: docker/compose:latest
      services:
        - docker
      caches:
        - docker
      script:
        - |
          <<'DC' docker-compose --file - up --abort-on-container-exit
          version: "3.8"
          services:
            minio:
              image: minio/minio:edge
              container_name: minio
              expose:
                - "9000"
              environment:
                MINIO_ACCESS_KEY: minio
                MINIO_SECRET_KEY: minio123
              command: server /data
            mc:
              image: minio/mc:latest
              depends_on:
                - minio
              entrypoint: >
                /bin/sh -c "/usr/bin/mc config host rm local
                            /usr/bin/mc config host add --quiet --api s3v4 \
                              local http://minio:9000 minio minio123
                            test `curl -o /dev/null -sw '%{http_code}' \
http://minio:9000` = 403 || sleep 1 \
                              && test `curl -o /dev/null -sw '%{http_code}' \
http://minio:9000` = 403 || sleep 1
                            /usr/bin/mc rb --force local/ciip-support/
                            /usr/bin/mc mb --quiet local/ciip-support/"
          DC
BBPL

(yes that is a Docker Compose document inside a Bitbucket Pipelines document); output:

 +++ step #1

name...........: "docker compose example"
effective-image: docker/compose:latest
container......: pipelines-1.docker-compose-example.default.pipelines
container-id...: 034ac6522b86

 +++ copying files into container...


 + <<'DC' docker-compose --file - up --abort-on-container-exit
version: "3.8"
services:
  minio:
    image: minio/minio:edge
    container_name: minio
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server /data
  mc:
    image: minio/mc:latest
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "/usr/bin/mc config host rm local
                  /usr/bin/mc config host add --quiet --api s3v4 \
                    local http://minio:9000 minio minio123
                  test `curl -o /dev/null -sw '%{http_code}' \
http://minio:9000` = 403 || sleep 1 \
                    && test `curl -o /dev/null -sw '%{http_code}' \
http://minio:9000` = 403 || sleep 1
                  /usr/bin/mc rb --force local/ciip-support/
                  /usr/bin/mc mb --quiet local/ciip-support/"
DC

Starting minio ... done
Recreating app_mc_1 ... done
Attaching to minio, app_mc_1
mc_1 | Removed `local` successfully.
mc_1 | Added `local` successfully.
minio | Endpoint: http://172.20.0.2:9000 http://127.0.0.1:9000
minio | 
minio | Browser Access:
minio | http://172.20.0.2:9000 http://127.0.0.1:9000
minio | 
minio | Object API (Amazon S3 compatible):
minio | Go: https://docs.min.io/docs/golang-client-quickstart-guide
minio | Java: https://docs.min.io/docs/java-client-quickstart-guide
minio | Python: https://docs.min.io/docs/python-client-quickstart-guide
minio | Javascript&colon; https://docs.min.io/docs/javascript-client-quickstart-guide
minio | .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
mc_1 | Removed `local/ciip-support/` successfully.
mc_1 | Bucket created successfully `local/ciip-support/`.
app_mc_1 exited with code 0
Stopping minio ... done
Aborting on container exit...

Example Notes:

• The Docker Compose document has been re-constructed from the code-block in your question
• The "mc" container needs to wait for minio to be healthy. Added a test with curl which was not part of your original question (same might apply to the first example)

This second example shows that it is entirely possible to just create a pipeline and let run docker compose in it. In such a scenario, you would not need to port the Docker Compose YAML to Bitbucket Pipelines YAML.

Some limitations might apply to docker compose in bitbucket pipelines, which can be due to security settings, but those are in the details and this should not stop you using docker compose in Bitbucket Pipelines.