Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Celebration

Earn badges and make progress

You're on your way to the next level! Join the Kudos program to earn points and save your progress.

Deleted user Avatar
Deleted user

Level 1: Seed

25 / 150 points

Next: Root

Avatar

1 badge earned

Collect

Participate in fun challenges

Challenges come and go, but your rewards stay with you. Do more to earn more!

Challenges
Coins

Gift kudos to your peers

What goes around comes around! Share the love by gifting kudos to your peers.

Recognition
Ribbon

Rise up in the ranks

Keep earning points to reach the top of the leaderboard. It resets every quarter so you always have a chance!

Leaderboard

Come for the products,
stay for the community

The Atlassian Community can help you and your team get more value out of Atlassian products and practices.

Atlassian Community about banner
4,463,578
Community Members
 
Community Events
176
Community Groups

Cache pip doesn''t work on bitbucket-pipeline

Edited

Hello everyone, I'm having a problem creating the project pipeline in which I participate. Due to the number of steps we have, it is necessary to use the cache, however the cache does not respond correctly as I imagined, not returning the list of saved packages so that they can be called and executed in later steps.

 

A clear example of this, we have a project that makes use of the Django framework and linter in some steps in parallel and after installing the packages they are not found breaking the flow.

 

Could help me with this problem?

 

image: python:3.10-slim

pipelines:

default:

- step:

name: Install requirements

caches:

- pip

script:

- pip install --require-hashes -r requirements/dev.txt

- parallel:

- step:

name: Flake8

caches:

- pip

script:

- pip install flake8

- flake8 --verbose --indent-size=4 --max-line-length=79 --ignore="E712,E24,W504,W503,E203,E501,F405" --select="C,E,F,W,B,B9"

(...)

- step:

name: Unittest && Coverages

caches:

- pip

script:

- export $(xargs < env.example)

- pip install --require-hashes -r requirements/dev.txt

- python manage.py test --settings=core.settings.tests

- coverage xml -i

artifacts:

- coverage.xml

 

Note:

  • I've already tried clearing the global cache, based on some suggestions from the community and recreated the cache, still with no effect.
  • Currently it is necessary to install packages in all steps for the pipeline to complete correctly!

 

error_caching_02.pngerror_caching_01.png

 

1 answer

0 votes

Hi @Renan Penna and welcome to the community.

I think that you need to include in your yml file the commands that install the tools/dependencies you are using.

  • The first time that a build completes successfully, the cache will be saved.
  • When another build runs, and if the cache is available, the installation command will use the cache instead of downloading the dependencies from the internet.

You will be able to see this e.g. in the output of the command pip install flake8

When no cache is available, the output will show

Collecting flake8
Downloading flake8-4.0.1-py2.py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.1/64.1 KB 12.6 MB/s eta 0:00:00
Collecting pycodestyle<2.9.0,>=2.8.0
Downloading pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.1/42.1 KB 6.9 MB/s eta 0:00:00
Collecting pyflakes<2.5.0,>=2.4.0
Downloading pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 69.7/69.7 KB 15.8 MB/s eta 0:00:00
Collecting mccabe<0.7.0,>=0.6.0
Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)

In a subsequent build, when the cache is available, the output will show

Collecting flake8
Using cached flake8-4.0.1-py2.py3-none-any.whl (64 kB)
Collecting pycodestyle<2.9.0,>=2.8.0
Using cached pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
Collecting mccabe<0.7.0,>=0.6.0
Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting pyflakes<2.5.0,>=2.4.0
Using cached pyflakes-2.4.0-py2.py3-none-any.whl (69 kB)

Please keep in mind that caches are saved for a week. Any cache which is older than 1 week will be cleared automatically and repopulated during the next successful build, so your builds should be configured to work whether or not the cache is present.

Kind regards,
Theodora

Hi @Theodora Boudale, how are you?

Thank you very much for the suggestion, but we already do the installation in the first step, its name is "Install requirements". In it we get all the packages including the packages for the later steps. And it still doesn't work.

Do I need to make explicit the installation of all packages for it to work?

Hi @Renan Penna,

Every step in Bitbucket pipelines runs in a separate Docker container, which gets destroyed once the step finishes.
Any installation you make e.g. in the first step, will be made on the Docker container of the first step only.
For subsequent steps, a new Docker container will start for each step.

If the tools that you need are not available in the Docker image that you use as a build container (python:3.10-slim in your case), then you will need to install them in the steps that use them.

Another thing you could do is create a custom Docker image where you install all the tools you need and use that as a build container, instead of python:3.10-slim:

Kind regards,
Theodora

Suggest an answer

Log in or Sign up to answer
TAGS

Atlassian Community Events