Why is my pip cache not found?

[nhoening]

Hello, I am trying to use pip caching in my pipeline and I believe I've followed the instructions here.

But my pipeline keeps logging (yes, on subsequent builds, as well):

Cache "pip": Downloading
Cache "pip": Not found

Here is my bitbucket-pipelines.yml:

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

Did I miss anything? Thanks!

[lassian]

Hi Nicolas,

Are you expecting the pip cache to be cretaed by the first step and consumed by the second? From looking at your script commands it appears that way (as the first step installs some pip dependencies that you want cached and the second runs tests that i assume use said dependencies).

If so you need to add the cache pip to the first step so it will generate it, so that the second step can consume it (and every other execution of that second step for the week will aswell :))

Here is the correct yaml that will generate the pip cache via the first step such that the second step can consume it.

pipelines:
  default:
    - step:
        name: Lint by Flake8
        image: python:3.6
        caches:
          - pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

Kind Regards,

Nathan Burrell 

[nhoening]

Hi Nathan,

thanks for your reply. I was under the impression that the cache would work in between builds, but you seem to suggest that the cache only works within a build. That would certainly clear this up.

What I don't quite get then is what you mean by "every other execution of that second step for the week". So the cache is stored for a week?

-Nicolas

[lassian]

Hi Nicolas,

In the first yaml example you provided.

pipelines:
  default:
    # This step installs dependencies that should be cached its however not configured
    # with a pip cache so it is not generating one.
    - step:
        name: Lint by Flake8
        image: python:3.6
        script:
            - pip install flake8-bugbear flake8
            - flake8 bvp
    # This step consumes the dependencies installed by the first step however as the
    # first step isnt configured with a pip cache you keep getting cache not found.
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres

The first step is installing dependencies via pip but it is not producing a pip cache as the step is not configured with the pip cache. Adding pip as a cache to a step designates it as both producing AND consuming the dependency. The second step was configured with the pip cache but it was not installing any dependencies and is why the pip cache was not being found.

In the second yaml I provided

pipelines:
  default:
    # This step installs dependencies and only ONCE a week (or if a user expires it manually)
    # will it upload a pip cache containing those dependencies.
    - step:
        name: Lint by Flake8
        image: python:3.6
        caches:
          - pip
        script:
          - pip install flake8-bugbear flake8
          - flake8 bvp
    # This step will now download and consume the dependencies cached by the first step of
    # the first pipeline that ran after the cache expired (either weekly or manually).
    - step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - python setup.py test
        services:
          - postgres 

The first step is now configured with a pip cache which also designates it will generate the pip cache if one doesnt exist. And as the second step is configured with the pip cache it will download it and consume it.

Which is how caches work within a build.

Now as to how caches work between builds and is what I mean by if a cache is found and also "every other execution of that second step for the week" is that once a week we expire caches, so that if users add any new dependencies they wont be added to the cache for that week, but they will be added to the new cache generated the week after.

In your example your first step only generates a new cache if one has not been generated for that week, and its only generated by the first step once a week. Everytime you execute that pipeline for that week the first step will download the pre-existing pip cache and it wont generate it again, and the second still will download it and consume it.

You can also manually expire these caches earlier or see the existence of the caches from the pipelines page under the cache setting.

For more information about caching please see: https://confluence.atlassian.com/bitbucket/caching-dependencies-895552876.html

Kind Regards,

Nathan Burrell

[nhoening]

Hi Nathan, after re-reading this now, I realise you thought my second step isn't installing anything, as there is no `pip install ...` anywhere. It is though, `python setup.py test` will install everything needed to run (all my dependencies for my code, which are quite a lot).

So, both of my steps install some python packages, but I have no pip cache. If I click on "chaches" in my pipeline home page, it always says "You haven't added any dependency caches yet".

Maybe (a) pip cache doesn't pick that up without the word "pip" being in the step description or (b) python's setuptools isn't actually using pip when doing it this way (`python setup.py <some-command>`) - those would be two explanations for the cache never ever being created.

The first explanation you guys would know about, I couldn't. I'm about to try pip-installing explicitly via a requirements.txt file before the tests run, to see if the cache will be created then. I'll report back :)

[nhoening]

Now I get some caching going, but after three pipeline runs, I still see this, indicating that basically nothing is actually cached:

Cache "pip": Downloading
Cache "pip": Downloaded 245.3 KiB in 1 seconds
Cache "pip": Extracting
Cache "pip": Extracted in 0 seconds

Every line in the pip install step begins with

Downloading https://files.pythonhosted.org/packages

My requirements.txt reads "-e ." btw

Here is my pipeline step:

- step:
        name: Build and Test
        image: python:3.6
        caches:
          - pip
        script:
          - ci/SETUP.sh
          - apt-get update
          - apt-get -y install postgresql-client
          - PGPASSWORD=a1test psql -h localhost -p 5432 -U a1test a1test;
          - pip install -r requirements.txt
          - python setup.py test
        services:
          - postgres

[Dan Frankowski]

I have the same problem. My step looks like:

pipelines:
  default:
    - step:
      caches:
        - pip
      script:
        - pip install -r requirements.dev.txt
        - ./manage.py test

Yet the step always has an empty cache.

Cache "pip": Downloading
Cache "pip": Not found

Again, I want caching *between* runs.

[Mike Medved]

One thing I noticed is that it appears the caching only happens on a successful run of the pipeline.  So if your cache was deleted (manually, or timed out once per week), you won't have cache restoring there until at least one successful pipeline run.

[Denise Skidmore]

Not a complete solution, but things to look for in the log

different cache, but possible issue with pip as well:
You already have a 'dotnetcore' cache so we won't create it again
Cache "dotnetcore: ~/.nuget/packages": Skipping upload for existing cache

does it only upload to the cache the first time the cache is referenced until that cache expires?

deleting my pip cache is triggering a new cache assembly, but the size is over limit so it doesn't upload:
Assembling contents of new cache 'pip'
You already have a 'dotnetcore' cache so we won't create it again
Cache "pip: ~/.cache/pip": Compressing
Cache "pip: ~/.cache/pip": Compressed in 146 seconds
Cache "pip: ~/.cache/pip": Compressed cache size is 2 GiB over the 1 GiB upload limit, so the cache will not be uploaded.

I'm currently working on dividing my requirements.txt up, and having an explicit cache creation step happen first, so that a limited set of packages are cached that are needed in multiple steps and are under the 1 GiB limit.