You're on your way to the next level! Join the Kudos program to earn points and save your progress.
Level 1: Seed
25 / 150 points
1 badge earned
Challenges come and go, but your rewards stay with you. Do more to earn more!
What goes around comes around! Share the love by gifting kudos to your peers.
Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!
Join now to unlock these features and more
The atlassian/google-gke-kubectl-run docker image is used by the GKE pipe of Atlassian
When you use it as a pipe in a pipeline (see https://bitbucket.org/product/features/pipelines/integrations?p=atlassian/google-gke-kubectl-run), it's downloaded the first time.
If it's used again in the same step, it's kept from local cache.
But if it's in another step (or another run), it's downloaded again (which slows down the pipeline)
I thought adding a cache would be enough:
but it did not seem to help
Personally I played around with the caching but it was faster to simply download the fresh image. According to https://hub.docker.com/r/bitbucketpipelines/google-gke-kubectl-run/tags it's doubled in size from 2.1.0 354.54MB to its current 3.0.0 753.52MB. Maybe the bigger size is better to cache to a local hard drive, but if Dockerhub serves the image from SSD in global data warehouses it takes something like 12s, and Bitbucket serves cache from SSD and it takes around 16s, then what's the difference, really? Just get the clean image again if it's a fairly popular one. I assume that caching would make sense if you have a customised image, but I'm not seeing benefits for a public one.
Are you building a docker image in any of the steps that have a docker cache definition?
If I understand correctly, you are using the same pipe in different steps and pipelines in your bitbucket-pipelines.yml file, is that correct?
If it's the same pipe, do you use the same version of the pipe in all these different places?
Do you have a definition for the docker cache in every step using the pipe?
Here is an excerpt of the pipeline:
definitions:steps:- step: &buildname: 'Build'services:- dockercaches:- dockerartifacts:- (...)script:- (...)- docker build (...)- step: &push-docker-imagename: 'Push docker image to registry'(...)- step: &deployname: 'Deploy'deployment: XXXartifacts:download: falsescript:(...)- pipe: atlassian/google-gke-kubectl-run:2.2.0variables:(...)pipelines:default:- step: *build- step: *push-docker-image- step:<<: *deployname: "Deploy to Dev"deployment: dev- step:<<: *deployname: "Deploy to Staging"deployment: stagingtrigger: 'manual'
So we use the same pipe (same version) several times in the pipeline, and we run this pipeline many times a day.
Currently, it downloads the docker image atlassian/google-gke-kubectl-run:2.2.0 every time, which looks not optimal.
I initially added the cache "docker" on step "deploy" but it did not help.
As I explained in my other answer, I suspect it's simply because the cache was already made by the "build" step, and is never updated.
So I probably simply need to use a custom cache, with a different name
Thank you for the details. You are right, if a docker cache is saved from the docker image you are building, then no docker cache will be saved for the pipe.
You can define a second cache in this case.
I am posting below an example yml file with (1) a definition of a custom cache for the image you are building and the commands you need to use to generate and load the cache and (2) a docker cache definition for the pipe.
I used your excerpt but removed some definitions like artifacts and some steps for simplicity:
- step: &build
- docker load -i docker-image-cache/* || echo "No cache"
- docker build .
- mkdir -p docker-image-cache && docker save $(docker images -aq) -o docker-image-cache/cache.tar
- docker image ls -aq
- step: &deploy
- pipe: atlassian/google-gke-kubectl-run:2.2.0
- step: *build
- step: *deploy
Make sure to use the following for every step that is using pipe: atlassian/google-gke-kubectl-run:2.2.0
Remove also any saved caches when you make the changes I suggested, so that the two caches will get populated for the image you build and for the pipe.
I hope this helps, if you have any questions, please feel free to let me know.
Thinking again about that, it might be because I already had a populated docker cache.
From what I understood, a bitbucket pipeline cache is never updated. It's only cleared after one week.
It might be the reason why it was not efficient here.
Maybe I should create a custom cache, with a different name, based on https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/#Custom-caches-for-other-build-tools-and-directories ?