nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why is my pip cache not found?

nhoening

Hello, I am trying to use pip caching in my pipeline and I believe I've followed the instructions here.

But my pipeline keeps logging (yes, on subsequent builds, as well):

Cache "pip": Downloading
Cache "pip": Not found

Here is my bitbucket-pipelines.yml:

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

Did I miss anything? Thanks!

Accepted answers

All comments

lassian

Hi Nicolas,

Are you expecting the pip cache to be cretaed by the first step and consumed by the second? From looking at your script commands it appears that way (as the first step installs some pip dependencies that you want cached and the second runs tests that i assume use said dependencies).

If so you need to add the cache pip to the first step so it will generate it, so that the second step can consume it (and every other execution of that second step for the week will aswell )

Here is the correct yaml that will generate the pip cache via the first step such that the second step can consume it.

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
        caches:
          - pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

Kind Regards,

Nathan Burrell

nhoening

Hi Nathan,

thanks for your reply. I was under the impression that the cache would work in between builds, but you seem to suggest that the cache only works within a build. That would certainly clear this up.

What I don't quite get then is what you mean by "every other execution of that second step for the week". So the cache is stored for a week?

-Nicolas

lassian

Hi Nicolas,

In the first yaml example you provided.

pipelines:
  default:
    # This step installs dependencies that should be cached its however not configured
    # with a pip cache so it is not generating one.
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
    # This step consumes the dependencies installed by the first step however as the
    # first step isnt configured with a pip cache you keep getting cache not found.
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

The first step is installing dependencies via pip but it is not producing a pip cache as the step is not configured with the pip cache. Adding pip as a cache to a step designates it as both producing AND consuming the dependency. The second step was configured with the pip cache but it was not installing any dependencies and is why the pip cache was not being found.

In the second yaml I provided

pipelines:
  default:
    # This step installs dependencies and only ONCE a week (or if a user expires it manually)
    # will it upload a pip cache containing those dependencies.
    - step:
        name: Lint by Flake8
        image: python:3.6
        caches:
          - pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
    # This step will now download and consume the dependencies cached by the first step of
    # the first pipeline that ran after the cache expired (either weekly or manually).
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

The first step is now configured with a pip cache which also designates it will generate the pip cache if one doesnt exist. And as the second step is configured with the pip cache it will download it and consume it.

Which is how caches work within a build.

Now as to how caches work between builds and is what I mean by if a cache is found and also "every other execution of that second step for the week" is that once a week we expire caches, so that if users add any new dependencies they wont be added to the cache for that week, but they will be added to the new cache generated the week after.

In your example your first step only generates a new cache if one has not been generated for that week, and its only generated by the first step once a week. Everytime you execute that pipeline for that week the first step will download the pre-existing pip cache and it wont generate it again, and the second still will download it and consume it.

You can also manually expire these caches earlier or see the existence of the caches from the pipelines page under the cache setting.

For more information about caching please see: https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html

Kind Regards,

Nathan Burrell

nhoening

Hi Nathan, after re-reading this now, I realise you thought my second step isn't installing anything, as there is no `pip install ...` anywhere. It is though, `python setup.py test` will install everything needed to run (all my dependencies for my code, which are quite a lot).

So, both of my steps install some python packages, but I have no pip cache. If I click on "chaches" in my pipeline home page, it always says "You haven't added any dependency caches yet".

Maybe (a) pip cache doesn't pick that up without the word "pip" being in the step description or (b) python's setuptools isn't actually using pip when doing it this way (`python setup.py <some-command>`) - those would be two explanations for the cache never ever being created.

The first explanation you guys would know about, I couldn't. I'm about to try pip-installing explicitly via a requirements.txt file before the tests run, to see if the cache will be created then. I'll report back

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

cloud

bitbucket

bitbucket-cloud

bitbucket-pipelines

cache