Pip cache not working for wxPython

[Jaswinder Singh]

Hi, 

Our repo has a dependency on wxPython. Due to wxPython not shipping a wheel, we have to compile it from source everytime we run a pipeline. I am looking for a way to cache the build information so that we don't have a 25 minute build time on every pipeline run. 

I thought setting up our yaml file like this would give us the desired result:

```

image: python:3.7.5

pipelines:

  default:

    - step:

        caches:

          - pip

        script: # Modify the commands below to build your repository.

          - apt update && apt install -y libgtk-3-dev

          - pip install pipenv

          - pipenv install

          - pipenv run python -m pytest

```

But our build times are still really long. 

[Patrik S]

Hello @Jaswinder Singh ,

Welcome to Atlassian Community.

I understand you are having issues to cache your python dependencies. Just to give you a background, caches in pipelines are basically saving the directory where your package manager/tool saves the dependency files in, and using it in the subsequent builds.

We have pre-defined caches for the most common languages and package-managers, such as pip, and they cache these languages' default cache directories. In case of pip, the default cache directory is ~/.cache/pip. You can check the directory for the other languages in the documentation below :

https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/#Pre-defined-caches

However, looking at your build you do not seem to be using pip to install the dependencies of your project, but is instead using a different package manager : pipenv. Accordingly to pipenv documentation, the cache is usually stored by default in the folder ~/.cache/pipenv, which is different from pip. We currently do not have a pre-defined cache for pipenv, but you can create your own custom cache pointing to pipenv's directory as the below example :

image: python:3.7.5
pipelines:
  default:
    - step:
        caches:
          - pipenv #cache is defined below in the definitions section
        script:
          - pipenv install
definitions:
  caches:
    pipenv: ~/.cache/pipenv

This will cache the  ~/.cache/pipenv directory for your subsequent pipeline runs.

Please note the defined cache will be saved after the first successful build. Also, caches last for 1 week, and once expired, they will be rebuilt in the next build (meaning the dependencies will be downloaded again from the internet).

Hope that helps to clarify your questions. Do let us now in case you run into any issues.

Thank you @Jaswinder Singh

Kind regards,

Patrik S

[Jaswinder Singh]

Hi Pratik, 

Thanks very much for taking the time to answer the question. I don't think we have arrived at the solution. I modified my pipelines file to use the cache for pipenv, and I am still getting the same 25 minute build times. 

Here is what the pipelines file currently looks like:

# This is a sample build configuration for Python.

# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.

# Only use spaces to indent your .yml configuration.

# -----

# You can specify a custom docker image from Docker Hub as your build environment.

image: python:3.7.5

pipelines:

  default:

    - step:

        caches:

          - pip

          - pipenv # cache defined below

        script: # Modify the commands below to build your repository.

          - apt update && apt install -y libgtk-3-dev

          - pip install pipenv

          - pipenv install

          - pipenv run python -m pytest

definitions:

  caches:

    pipenv: ~/.cache/pipenv

Here is the output from my build setup, where it is obviously pulling in the pipenv cache, 

Here is the output of my pipenv install step:

Any additional guidance is greatly appreciated. Thanks very much!

Jaswinder

[Patrik S]

Hey @Jaswinder Singh ,

From the build logs you have shared, the folder ~/.cache/pipenv actually had something, as it generated a 212.4 MiB cache file.

So the issue here seems to be that pipenv is not using that cache, or looking at somewhere else. In this case, this falls a bit outside of Pipelines scope, and is more related on how pipenv handles the cache, as pipeline is correctly generating the cache for the directory provided.

I would recommend you to test the cache itself by using a small repo with like 1 dependency to identify what it takes within pipenv to actually use the cache.

Additionally, pipenv documentation also provides an environment variable that you can use to change the cache location (PIPENV_CACHE_DIR). You can try changing the default directory by setting that environment variable, create a cache definition in your YML file for that same path, and check how it goes.

If it still does not work, I would say you could try using a different package manager, like pip, or try doing a deeper investigation on how pipenv cache works and which are all the folders it looks for, so you can cache all the required directories in your build.

Thank you, @Jaswinder Singh .

Kind regards,

Patrik S