Caching poetry on Bitbucket Pipelines

[Matteo Verità]

I'm trying to setup my bitbucket_pipelines.yml file to make it possible for all the steps to have Poetry already installed, so that running `poetry run [command]` should never return `bash: poetry: command not found`. So far I made it work with the following configuration:

# This is an example Starter pipeline configuration
# Use a skeleton to build, test and deploy using manual and parallel steps
# -----
# You can specify a custom docker image from Docker Hub as your build environment.

image: python:3.12

definitions:
  caches:
    poetry-cache:
      key:
        files:
          - poetry.lock
      path: .venv

pipelines:
  default:
    - parallel:
      - step:
          name: 'Build and Test'
          caches:
            - poetry-cache
          script:
            - pipe: atlassian/poetry-cli-setup:0.2.0
            - ./setup-poetry.sh
            - poetry config virtualenvs.in-project true
            - poetry install
            - poetry run pytest
      - step:
          name: 'Lint'
          caches:
            - poetry-cache
          script:
            - pipe: atlassian/poetry-cli-setup:0.2.0
            - ./setup-poetry.sh
            - poetry install
            - poetry run ruff check

As you can see, the only way I found was to reinstall Poetry in each step. It's already good enough that `poetry install` is fast because it uses the cache and in fact ends up installing nothing, however I wonder if there's a better way for poetry itself to make it available to all steps without having to install it in each.

[Joshua Tang · Accepted solution]

Hi Matteo,

We have just released a new version of poetry-cli-setup:0.2.2 with some changes to allow caching of the Poetry installation. Similar to how you're caching the virtual environment, you can define an extra cache for the Poetry installation:

definitions:
  caches:
    poetry-cli-cache:
      key:
        files:
          - bitbucket-pipelines.yml
      path: /opt/poetry

Then you can include this cache in your individual steps, where the Poetry setup step would reuse the cached installation:

- step:
    name: 'Lint'
    caches:
      - poetry-cache
      - poetry-cli-cache
    script:
      - pipe: atlassian/poetry-cli-setup:0.2.2
      - ./setup-poetry.sh
      - poetry install
      - poetry run ruff check

This example is also documented on the README of the pipe: https://bitbucket.org/atlassian/poetry-cli-setup/src/master/

Hope this helps!

Kind regards,
Joshua

[Matteo Verità]

Thank you @Joshua Tang! It worked splendidly.

And thanks @Ben for the additional help!

[jbsilva]

Could you provide a full example?
When I removed `./setup-poetry.sh` from my second step I got `bash: poetry: command not found`.

[Ben]

Hi @jbsilva 

Please refer to the pipe documentation for a full example, the error message you've received generally means that the command you are executing can't be executed as a setup step is missing (ie the bash script you removed):
https://bitbucket.org/atlassian/poetry-cli-setup/src/master/

Cheers!

- Ben (Bitbucket Cloud Support)

[jbsilva]

By full example I mean one with at least two steps.

[Ben]

Hi Matteo,

You can potentially use artifacts to achieve this - so basically you could figure out where poetry is installed in the context of your build, then compress it as a zip - pass it to subsequent steps as an artifact, unzip it and execute poetry commands:

• https://community.atlassian.com/t5/Bitbucket-questions/Is-there-a-way-of-sharing-data-between-steps-in-bitbucket/qaq-p/777977

Other than that, I have reached out to our pipes development team to figure out the best place to raise suggestions.

Cheers!

- Ben (Bitbucket Cloud Support)