Forums

Articles
Create
cancel
Showing results for 
Search instead for 
Did you mean: 

Cache Python pip install from docker build

lcucurachi May 29, 2025

Hello, I'm building a docker image and inside the Dockefile I'm doing:

 

RUN --mount=type=cache,target=/root/.cache/pip \
        pip install -r requirements.txt

While in the pipeline for the step I have something like:

 

- step: &build-app-image
        name: Build app image
        caches:
          - docker
          - pip

        script:
          - docker build . --file Dockerfile --tag ${IMAGE}

I've noticed that every time the `pip install` during the build is downloading the libraries all the time. So I believe the caching does not work.

I don't understand why. I've tried all kinds of things.

For example, doing a bind mount with rw permissions, but it then complains that `/root/.cache/pip` doesn't exist but it does because BitBucket pipeline is storying a pip cache and I've already checked that `ls /root/.cache/pip` returns something. 

Can anyone step in to help please? In theory what I'm doing (without bind mount but with cache mount) should work perfectly fine.

 

I think the docker cache not working is exactly reported here as my builds always start from scratch and do not seem to cache any of the build steps.

-  https://jira.atlassian.com/browse/BCLOUD-23193

1 answer

1 vote
Jim Knepley - ReleaseTEAM
Atlassian Partner
May 29, 2025

Welcome to the community, @lcucurachi 

For that cache to be meaningful, your Bitbucket runners must be able to access the host filesystem, and future runs should be on the same host. Using cloud runners, I think caching like that is doomed from the start.

You might be able to pull something off if you're using self-hosted runners, but I wouldn't advise using runners that keep any files between runs. Completely ephemeral runners are the current best practice.

If you're using a Python venv or similar, you build the venv, .zip it up, and use it as an artifact. Just download the .zip and decompress it during the build instead of using pip.

ref: https://support.atlassian.com/bitbucket-cloud/docs/runners/ ("Linux Docker-only")

lcucurachi May 30, 2025

Thanks, that makes sense.  It sounds like my only option is to use an external cache using the `--cache-from` and `--cache-to` commands to overcome this issue. Not the best solution but I will try. And it doesn't help that the docker layer caching is broken. 

lcucurachi May 30, 2025

The caching feature of build layers has been deprecated in RuntimeV3 while in the previous version it doesn't work at all. I think this is quite sad since it's a very important feature.

Suggest an answer

Log in or Sign up to answer
DEPLOYMENT TYPE
CLOUD
PRODUCT PLAN
PREMIUM
TAGS
AUG Leaders

Atlassian Community Events