Cache pip doesn''t work on bitbucket-pipeline

[Renan Penna]

Hello everyone, I'm having a problem creating the project pipeline in which I participate. Due to the number of steps we have, it is necessary to use the cache, however the cache does not respond correctly as I imagined, not returning the list of saved packages so that they can be called and executed in later steps.

A clear example of this, we have a project that makes use of the Django framework and linter in some steps in parallel and after installing the packages they are not found breaking the flow.

Could help me with this problem?

image: python:3.10-slim

pipelines:

   default:

     - step:

         name: Install requirements

   caches:

      - pip

   script:

      - pip install --require-hashes -r requirements/dev.txt

    - parallel:

      - step:

         name: Flake8

         caches:

            - pip

         script:

            - pip install flake8

            - flake8 --verbose --indent-size=4 --max-line-length=79 --ignore="E712,E24,W504,W503,E203,E501,F405" --select="C,E,F,W,B,B9"

       (...)

       - step:

         name: Unittest && Coverages

         caches:

           - pip

         script:

            - export $(xargs < env.example)

            - pip install --require-hashes -r requirements/dev.txt

            - python manage.py test --settings=core.settings.tests

            - coverage xml -i

         artifacts:

            - coverage.xml

Note:

• I've already tried clearing the global cache, based on some suggestions from the community and recreated the cache, still with no effect.
• Currently it is necessary to install packages in all steps for the pipeline to complete correctly!
  
  

[Theodora Boudale]

Hi @Renan Penna and welcome to the community.

I think that you need to include in your yml file the commands that install the tools/dependencies you are using.

• The first time that a build completes successfully, the cache will be saved.
• When another build runs, and if the cache is available, the installation command will use the cache instead of downloading the dependencies from the internet.

You will be able to see this e.g. in the output of the command pip install flake8

When no cache is available, the output will show

Collecting flake8
Downloading flake8-4.0.1-py2.py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.1/64.1 KB 12.6 MB/s eta 0:00:00
Collecting pycodestyle<2.9.0,>=2.8.0
Downloading pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.1/42.1 KB 6.9 MB/s eta 0:00:00
Collecting pyflakes<2.5.0,>=2.4.0
Downloading pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 69.7/69.7 KB 15.8 MB/s eta 0:00:00
Collecting mccabe<0.7.0,>=0.6.0
Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)

In a subsequent build, when the cache is available, the output will show

Collecting flake8
Using cached flake8-4.0.1-py2.py3-none-any.whl (64 kB)
Collecting pycodestyle<2.9.0,>=2.8.0
Using cached pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
Collecting mccabe<0.7.0,>=0.6.0
Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.5.0,>=2.4.0
Using cached pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)

Please keep in mind that caches are saved for a week. Any cache which is older than 1 week will be cleared automatically and repopulated during the next successful build, so your builds should be configured to work whether or not the cache is present.

Kind regards,
Theodora

[Renan Penna]

Hi @Theodora Boudale, how are you?

Thank you very much for the suggestion, but we already do the installation in the first step, its name is "Install requirements". In it we get all the packages including the packages for the later steps. And it still doesn't work.

Do I need to make explicit the installation of all packages for it to work?

[Theodora Boudale]

Hi @Renan Penna,

Every step in Bitbucket pipelines runs in a separate Docker container, which gets destroyed once the step finishes.
Any installation you make e.g. in the first step, will be made on the Docker container of the first step only.
For subsequent steps, a new Docker container will start for each step.

If the tools that you need are not available in the Docker image that you use as a build container (python:3.10-slim in your case), then you will need to install them in the steps that use them.

Another thing you could do is create a custom Docker image where you install all the tools you need and use that as a build container, instead of python:3.10-slim:

• https://support.atlassian.com/bitbucket-cloud/docs/use-docker-images-as-build-environments#Custom-build-environments
• https://support.atlassian.com/bitbucket-cloud/docs/use-docker-images-as-build-environments#Creating-a-custom-build-environment

Kind regards,
Theodora