How can one add build steps to a docker image before the pipeline steps?

[RLogik]

As part of my build process, I need to pull a python image, then run a few commands (install dos2unix, etc. via apt-get, change a few values in files). Essentially I run the equivalent of a Dockerfile-build script.

Obviously, I can do these inside the pipeline, but the problem is, it would be an absolute nightmare to figure out what artefacts (considering I'm installing things) to declare.

Hence I need to run these as preliminary build steps, upon which the rest of the pipeline is to build on.

I this possible. If so how do I do this?

[ktomk · Accepted solution]

Hello @RLogik , welcome to the Atlassian Community.

If I understand your question the way that you find it cumbersome to maintain the preliminary build operations in a step scripts (e.g. many lines), one solution is to create a shell script and commit it to the repository and call it at the beginning of the step script.

Another solution which is less flexible is to create your own build image (as you wrote, that is like a Dockerfile). Keep in mind as this is less flexible this is more for the time when the build process has settled and it is clear what there is to encode. Then having a dedicated build image will show its benefits well.

You show little in your question about your concrete scenario, so I hope this already matches with your needs. I had problems to understand the meaning of artifacts, are those the artifacts in the Bitbucket Pipeline YAML file format?

[RLogik]

This is about the bitbucket-pipelines.yml script. For example:

image:
  name: <some docker image like python:xyz>
  # here I want to *additionally* run ONCE and for all build steps,
  # as I do in my own Dockerfile.
  # This should result in an image (let's call it main:base),
  # with which ALL the following steps are to work with
  # (instead of python:xyz).
  # These build steps are NOT simply "write a string to these few files".
  # They're apt-get install ... and curl instructions etc.
  # which install who knows what files and where.
  # So there is no way I can do this below as a step, and then copy artefacts.
pipelines:
  default:
    - step:
        name: Code build step
        script:
            - ....
    - step:
        name: Unit tests
        script:
            - ....
    - step:
        name: Compilation
        script:
            - ....

I know I can pack things in bash scripts. And I already do this. But that does not address the key issue. At the moment the only solution I have is to get the image, and then do everything in one single step (i. e. the docker build steps, code build, unit test, etc.), which works, but it is quite silly to have a pipeline with one step.

[ktomk]

This is a good example, makes it much more clear.

This looks like looking for a build image, most fitting an intermediate one.

I know for my local pipelines runner it works straight forward as one step can build a docker container and anther step can re-use it. But this is on a local machine.

On Bitbucket this needs some extra work like pushing it to a remote docker image repository (here Docker Hub), which requires some setup like credentials and authentication:

image:
   name: vendor/main:base
definitions:
   main-base: *build-image
     name: Build Main Base
     services:
       - docker
     caches:
       - docker
     image: python:alpine
     script:
       - TAG=vendor/main:base
       - |
         <<'DOCKERFILE' docker build --build-arg FROM=python:alpine -t "$TAG" -
         ARG FROM
         FROM $FROM
         # ... base image modifications for main:base
         DOCKERFILE
       - docker login -u "$DOCKER_HUB_USER" -p "$DOCKER_HUB_PASSWORD"
       - docker push "$TAG"
pipelines:
  default:
    - step: &build-image
    - step:
        name: Code build step
        script:
          - echo ....
     - step:
         name: Unit tests
         script:
           - echo ....
      - step:
          name: Compilation
          script:
            - echo ....

Technically this work on Bitbucket, tested it.

This will have side-effects on multiple pipelines running in parallel as the build image can change while the build is running.

It is perhaps better to have one pipeline to build the main:base image and push it to the registry so other pipeline steps can just use it without taking care about the building.

You can then re-build and push the main:base image in regular intervals to keep it updated by scheduling that pipeline.

[ktomk]

And technically, if all three steps are using the same image, there is not really a need to have them in multiple steps. normally the image is the boundary of the step, as only per step the image can be specified.

Maybe you can give some more context what is stupid in your eyes of having all in one step. Just asking.

[RLogik]

Thanks for your reply, @ktomk . Your suggestion is very useful. To answer your second question: I'm used to working with jenkins pipelines and breaking down a build into multiple steps (e.g. build, unit tests, compilation, e2es, distribution, depending upon the project).

Your trick of referencing an image and the creating that image as the first step is really neat. It seems to be the right way. I'm getting an error at the moment.

Running

image:
  name: vendor/main:basedefinitions:
  main-base: &build-image
    name: Build Main
    Base services:
      - docker
    caches:
      - docker
    image: ubuntu
    script:
      - TAG=vendor/main
      - |
      <<'DOCKERFILE' docker build --build-arg FROM=ubuntu -t $TAG
      ARG FROM
      FROM $FROM
      ARG DEBIAN_FRONTEND=noninteractive
      RUN apt-get -y update 2> /dev/null >> /dev/null
      RUN apt-get install -y dos2unix 2> /dev/null >> /dev/null
      COPY . /usr/apps/eis
      WORKDIR /usr/apps/eis
      # RUN . entry.sh # skipping, but I do something like this
      RUN echo 'Hello world!' >| MESSAGE_FILE DOCKERFILE
      - docker login -u "$DOCKER_HUB_USER" -p "$DOCKER_HUB_PASSWORD"
      - docker push "$TAG"
pipelines:
  default:
    - step: *build-image
    - step:
        name: Step One
        script:
          - bash [-f MESSAGE_FILE] # should succeed!
          - cat MESSAGE_FILE # should print 'Hello world!' to console
          - echo 'Hello Mars!' >| NEW_MESSAGE_FILE
        artifacts: # ignore wrong spelling
          - NEW_MESSAGE_FILE
    - step:
        name: Step Two
        script:
          - bash [-f NEW_MESSAGE_FILE] # should succeed!
          - cat NEW_MESSAGE_FILE # should print 'Hello Mars!' to console

(That took ages to copy in, Bitbucket won't let me copy in text with returns and indentation!!)

I get the following error message:

So I did the build step wrong. Perhaps you could look over this and tell me what I did wrong?

Remark: I actually have Dockerfile + .env + docker-compose.yml in my repository. Locally on my machine, I start docker with docker-compose and build+test my project (which is what the pipeline is intended to automate upon pushing). Is there a way to use my repository's Dockerfile in the "Build Main" definition?

Thanks again for your help!

EDIT: I now changed the main-base definition to

  main-base: &build-image
    name: Build Main Base
    services:
      - docker
    caches:
      - docker
    image: ubuntu
    script:
      - docker build -t ubuntu .
      - docker login -u "$DOCKER_HUB_USER" -p "$DOCKER_HUB_PASSWORD"
      - docker push "vendor/main:base"

That now builds correctly from the repository's Dockerfile. However, the "docker login" bit is where it now catches. I get a `Cannot perform an interactive login from a non TTY device` Error. How do I run this bit non-interactively?

[ktomk]

First the error, it's a missing parameter:

  ... FROM=ubuntu -t $TAG


should be:


  ... FROM=ubuntu -t $TAG -
                          ^
                     missing dash

And second, yes, you can use your Dockerfile. I would actually expect to make use of one as having the Dockerfile contents within the pipeline step script in form of a here-document is likely error-prone and brittle.

This was merely a quick hack from end for a proof of concept and it somehow continued well to the example of your this is build upon.

Just FYI, you can also run docker-compose in a pipeline.

Thanks for your reply on Jenkins, yes, that is perhaps a bit different, perhaps first look for a working version and then re-visit the structure and how you can ease the migration from Jenkins to Bitbucket Pipelines.

I think with this part in your Dockerfile you won't have much fun in Pipelines:

      COPY . /usr/apps/eis
      WORKDIR /usr/apps/eis

The pipelines it's the other way round. The project files are mounted into the build container and when the step script runs, the workdir is set to it.

It's perhaps really much easier you isolate creating the build image and just have it prepared instead of building it on each pipeline run as you are probably doing too much (probably, just saying. also it's fine in my eyes to play with these things).

And yes, copy and pasting is a mess for code blocks. No idea which profession this Atlassian company targets as it's audience ... ;)

[RLogik]

Thanks, fixed the build bit, and will keep that "COPY/WORKDIR" part in mind. Have edited my response. There's an issue now with the "push" step (cf. end of my editted reply). Any ideas?

[ktomk]

Hmm, the error message first made me thought "funny, I didn't had that issue yesterday". Thinking twice I have the following idea what is going on:

• login with username and password is tried
• it fails
• login on TTY (interactive terminal) is tried which must fail in pipelines as it is automated and asking for the user to enter the password will lead nowhere.

Double check the login works by verifying username and password in the environment variables (have you set them up on Bitbucket? I forgot to mention that requirement specifically as I thought it's known, but that perhaps was shortsighted. let me know if you need more details regarding environment variables on Bitbucket Pipelines).

If it helps (you need a little bit of setup for it as environment variables are involved, but it should be straight forward), I'm writing and running my pipelines first locally by making use of my pipelines utility. When things are setup I just push them to Bitbucket. This is much more comfortable and allows me to achieve much faster iterations. It's also a nice way to sandbox things, it works directly from the working tree, so does not even need to make commits to get things running. No idea if you've got a Linux or Mac box, don't know about Windows support.

Environment variables can be controlled with `.env` / `.env.dist` files for the pipelines utility. Let me know if it's helpful. And no strings attached. Everything the tool does you can do with docker itself and some shell scripting, it also shows what it does with the `-v` (verbose) switch.

Re-reading your last replies, and looking on my previous: The suggestion to have a dedicated build image still looks okay-ish to me, this is generally a good way with Bitbucket Pipelines. However I get the impression that you have everything already set-up in docker composer so it could be a better option in your project to make the pipelines docker-compose based.

Yesterday or the day before I just left a longer answer with two examples comparing both approaches. This might be insightful in your case as well:

• Answer to "What is the difference in syntax between bitbucket-pipelines services and docker-compose?"

[Tuan Nguyen Anh]

Hi , I'm not sure about this in pipeline but you can use "artifacts" in a step and use it for the next step.

[RLogik]

That's exactly what I want to avoid.