Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Docker build in pipelines doesn't cache build process?

Is there a way to cache build process for docker build?

Now, everytime the build is started, it downloads the parent image again and build goes through every Dockerfile command all over again, so the build in whole stretches up to 6 minutes.

 

Docker engine is capable of caching images and build commands that weren't changed.

I couldn't find any documentation on docker cashing ( found the ones for npm for example ). Is there a support for this but it's not documented ?  Or, if not, is this somewhere in the Pipelines roadmap?

 

THanks a lot!

 

1 answer

0 votes
SebC Atlassian Team Nov 09, 2017

Hey @msvoren,

Not at this time; follow https://bitbucket.org/site/master/issues/14144/cache-docker-layers-between-builds to get updates as we implement this feature.

 

-Seb

This feature has been shipped for customers to use here is the details on how to opt-in to this docker layer cache.

pipelines:
  default:
    services:
      - docker
    caches:
      - docker
    script:
      - # .. do cool stuff with Docker .. 

reference

Like Carlos_Ferreyra likes this

Hi @trevor,

Probably this feature does not correctly work, for example in my pipeline it caches only 2 layers:

Status: Downloaded newer image for ruby:2.5.0

---> 213a086149f6

Step 2/18 : ARG DATABASE_URL

---> Using cache

---> 429c813b6abf

Step 3/18 : ENV RACK_ENV production

---> Using cache

---> c0e98fc7cea0

Step 4/18 : ENV RAILS_ENV production

---> Running in 62f18ad58da3

Removing intermediate container 62f18ad58da3

---> a7f6146135a4

Step 5/18 : ENV DATABASE_URL $DATABASE_URL

---> Running in d0f15c5589fc

Removing intermediate container d0f15c5589fc 

 

Like # people like this

Bump! 

Did you get this to work?

Like Mohamed Bana likes this

I'm getting this same issue now.

Though the step that seems to never be cached is one where I run `apt-get update && apt-get install <stuff>`

Sending build context to Docker daemon 1.485MB
Step 1/24 : FROM php:7.3.9-apache
7.3.9-apache: Pulling from library/php
Digest: sha256:2f3a147ca94e87210d596ecb31533c127928bdb9cc4f2dc3364d80a3dd6024e4
Status: Downloaded newer image for php:7.3.9-apache
---> 6f7c5e29a126
Step 2/24 : ENV APACHE_DOCUMENT_ROOT /src/public
---> Using cache
---> 2594124b1607
Step 3/24 : RUN mkdir /src
---> Using cache
---> 4534159c5c9c
Step 4/24 : WORKDIR /src
---> Using cache
---> a4c50b69d084
Step 5/24 : RUN apt-get update && apt-get install -y unzip libpng-dev libjpeg62-turbo-dev locales apt-transport-https gnupg2
---> Running in 1c394017dd4a
trevor Atlassian Team Apr 06, 2020

When installing and doing an `apt-get update` likely these steps will never be cached as they fetch the metadata of the latest released packages before continuing.

@trevor but this is standard Docker stuff to do `apt-get update && apt-get install ...`, and caching of it has always worked fine when doing this locally in Docker outside of Bitbucket Pipelines.

Like # people like this

It's a horrible idea :( but im considering having "base" images which I then make the dynamic changes into since the build process takes "many minutes" when with caching it would take seconds.  

e.g.

FROM my-base-build-image

RUN do some dynamic stuff here

 

But ultimately this is a super poor way to do and isn't best practise.  Pipelines take so much longer than they should.

 

I guess the potential overheads of caching everyones stuff would be massive.   

We're having the same issue. All layers after a line with `apt-get update` and `apt-get install` are being fully rebuilt, always. It's not the default Docker behavior and even recommended to always combine the two on one line.

TiM I'm New Here Jun 30, 2020

well this is ridiculous, local docker build fully caches, yet in my pipeline with caches: docker it seems to do almost nothing resulting in full build which takes minutes :(


Like Jose Alban likes this

This is still happening. It will cache the `ADD yarn.lock` but then wont cache the `ADD Gemfile.lock`. 

Like Zogoo likes this

Any update? I think Bitbucket Docker cache has a very poor feature. Which is only allowed "cache: docker". And saying leave to the decision to Atlassian team.

So, It's not working for "RUN bundle install". 

Why you guys just allow to us what folder must be cached or not like normal pipeline cache?

Then at least please allow to us that we can choose a folder that we want to cache.

 

 

What's the status on this? @SebC @trevor 

Only a few arbitrary layers seem to be cached even with cache: docker.

A simple COPY package.json ./ fails the cache, even when building the same commit in a new build right after the first one.

IMHO this is unworthy of a professional building platform.

Like # people like this

Is it not updating the cache on every single pipeline build?

Is it creating a cache on the first build and not re populating it until the cache is expired?

I just looked at my caches in the top right of the pipelines build and the cache was created on the 23rd of September. We run builds every day, excluding the weekends, and I just now completed a build and the build cache has the same date on it.

That would definitely explain the behavior everyone is seeing.

If so that is definitely not how any other build cache works, and has to cost bitbucket more in build minutes than it does in replacing cache data.

and now I just saw this in the logs...

Skipping assembly of docker cache as one is already presentCache "docker": Skipping upload for existing cache

That has to be what's going on here. Why wouldn't Atlassian update the cache on every single successful build? What is the point of a week old cache? And why would the arbitrary build that happens to run right after the invalid cache be any more valid than any other? Currently it's being used like it's a thoughtfully curated cache.

Is there any way to force update the cache as part of the build?

Like # people like this

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Powering DevOps with Bitbucket Server & Data Center

Hi everyone, The Cloud team recently announced 12 new DevOps features that help developers ship better code, faster   ! While we’re all excited about the new improvements to Bitbucket ...

1,878 views 0 7
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you