Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Why is my pip cache not found?

Hello, I am trying to use pip caching in my pipeline and I believe I've followed the instructions here.

But my pipeline keeps logging (yes, on subsequent builds, as well):

Cache "pip": Downloading
Cache "pip": Not found

Here is my bitbucket-pipelines.yml:

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres


Did I miss anything? Thanks!

1 answer

0 votes

Hi Nicolas,

Are you expecting the pip cache to be cretaed by the first step and consumed by the second? From looking at your script commands it appears that way (as the first step installs some pip dependencies that you want cached and the second runs tests that i assume use said dependencies).

If so you need to add the cache pip to the first step so it will generate it, so that the second step can consume it (and every other execution of that second step for the week will aswell :))

Here is the correct yaml that will generate the pip cache via the first step such that the second step can consume it.

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
caches:
- pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

Kind Regards,

Nathan Burrell 

Hi Nathan,

thanks for your reply. I was under the impression that the cache would work in between builds, but you seem to suggest that the cache only works within a build. That would certainly clear this up.

What I don't quite get then is what you mean by "every other execution of that second step for the week". So the cache is stored for a week?

-Nicolas

Hi Nicolas,

In the first yaml example you provided.

pipelines:
  default:
# This step installs dependencies that should be cached its however not configured
# with a pip cache so it is not generating one.
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
# This step consumes the dependencies installed by the first step however as the
# first step isnt configured with a pip cache you keep getting cache not found.
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

The first step is installing dependencies via pip but it is not producing a pip cache as the step is not configured with the pip cache. Adding pip as a cache to a step designates it as both producing AND consuming the dependency. The second step was configured with the pip cache but it was not installing any dependencies and is why the pip cache was not being found.

In the second yaml I provided

pipelines:
  default:
# This step installs dependencies and only ONCE a week (or if a user expires it manually)
# will it upload a pip cache containing those dependencies.
    - step:
        name: Lint by Flake8
        image: python:3.6
caches:
- pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
# This step will now download and consume the dependencies cached by the first step of
# the first pipeline that ran after the cache expired (either weekly or manually).
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres 

The first step is now configured with a pip cache which also designates it will generate the pip cache if one doesnt exist. And as the second step is configured with the pip cache it will download it and consume it.

Which is how caches work within a build.

Now as to how caches work between builds and is what I mean by if a cache is found and also "every other execution of that second step for the week" is that once a week we expire caches, so that if users add any new dependencies they wont be added to the cache for that week, but they will be added to the new cache generated the week after.

In your example your first step only generates a new cache if one has not been generated for that week, and its only generated by the first step once a week. Everytime you execute that pipeline for that week the first step will download the pre-existing pip cache and it wont generate it again, and the second still will download it and consume it.

You can also manually expire these caches earlier or see the existence of the caches from the pipelines page under the cache setting.

For more information about caching please see: https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html

Kind Regards,

Nathan Burrell

Hi Nathan, after re-reading this now, I realise you thought my second step isn't installing anything, as there is no `pip install ...` anywhere. It is though, `python setup.py test` will install everything needed to run (all my dependencies for my code, which are quite a lot).

So, both of my steps install some python packages, but I have no pip cache. If I click on "chaches" in my pipeline home page, it always says "You haven't added any dependency caches yet".

Maybe (a) pip cache doesn't pick that up without the word "pip" being in the step description or (b) python's setuptools isn't actually using pip when doing it this way (`python setup.py <some-command>`) - those would be two explanations for the cache never ever being created.

The first explanation you guys would know about, I couldn't. I'm about to try pip-installing explicitly via a requirements.txt file before the tests run, to see if the cache will be created then. I'll report back :)

Now I get some caching going, but after three pipeline runs, I still see this, indicating that basically nothing is actually cached:

Cache "pip": Downloading
Cache "pip": Downloaded 245.3 KiB in 1 seconds
Cache "pip": Extracting
Cache "pip": Extracted in 0 seconds

Every line in the pip install step begins with

Downloading https://files.pythonhosted.org/packages

My requirements.txt reads "-e ." btw

Here is my pipeline step:

- step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - apt-get update
          - apt-get -y install postgresql-client
          - PGPASSWORD=a1test psql -h localhost -p 5432 -U a1test a1test;
          - pip install -r requirements.txt
          - python setup.py test
        services:
          - postgres

I have the same problem. My step looks like:

pipelines:
default:
- step:
caches:
- pip
script:
- pip install -r requirements.dev.txt
- ./manage.py test

Yet the step always has an empty cache.

Cache "pip": Downloading
Cache "pip": Not found

Again, I want caching *between* runs.

One thing I noticed is that it appears the caching only happens on a successful run of the pipeline.  So if your cache was deleted (manually, or timed out once per week), you won't have cache restoring there until at least one successful pipeline run.

Like # people like this

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

3,818 views 10 16
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you