How to configure the cache for the conda package manager in bitbucket pipelines

[David Michael Gang]

Hi all,

I use the following pipeline for checking the source code:

# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: continuumio/miniconda3:latest

pipelines:
  default:
    - step:
        script: # Modify the commands below to build your repository.
          - conda env update -n root --file  environment.yml
          - pylint module1
          - mypy -m module1 --ignore-missing-imports --strict
          - coverage run --source=module1 -m unittest discover
          - coverage report -m

I want to use the bitbucket caches to make the pipeline faster but i am not sure which directories i should cache.

I already asked the question at the conda user group   but did not get an answer.

How can i configure the cache ?

Thanks,

David

[Rajat Jain · Accepted solution]

image: python:3.6.2
pipelines:
  default:
    - step:
        caches:
          - condacache
        script:
          - /opt/python/bin/conda update -y conda
          - /opt/python/bin/pip install --upgrade pip
          - /opt/python/bin/pip install -r requirements.txt
          - /opt/python/bin/pytest

definitions:
  caches:
    condacache: /opt/python/bin


#This is how i used to create cache in my pipeline: Please take it as reference.
#This is a custom cache technique. Hope it would help you.

[tasubo]

This does not work.

[Denise Skidmore]

You can get the location of your package cache from the command: conda info

Add this to one of your steps and inspect the results, then you can define

your cache as in the other answers.

https://docs.anaconda.com/free/anaconda/packages/shared-pkg-cache/

https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/

[Michiel]

When using the `continuumio/miniconda3` image in your pipeline, I have found that defining a `conda` cache as `/opt/conda/pkgs` worked. After the first run of the pipeline, in subsequent runs of the pipeline the command `conda env update` should not longer print "Downloading and Extracting Packages"

[joan_massich]

This does not work for me.

image: continuumio/miniconda3:latest

pipelines:
  default:
    - step:
      caches:
        - conda
        - codatarballs

      script:
        - conda env update -n test_env --file environment.yml

definitions:
   caches:
     conda: ~/.conda/envs/test_env
     codatarballs: /opt/conda/pkgs

The output I get is:

Assembling contents of new cache 'conda'
Assembling contents of new cache 'codatarballs'

but then on the next run:


Cache "conda": Downloading
Cache "conda": Not found
Cache "codatarballs": Downloading
Cache "codatarballs": Not found

[Michiel]

I don't know why it doesn't download the cache in the next run, it does for me, and I have the same output at the end of the first run:

Assembling contents of new cache 'conda'

One thing you can check is in the main pipelines screen of the repository, there should be a button on the top right named "Caches". If I click that I see:

conda: opt/conda/pkgs 350.31 MB 29 May 2020

Hope this helps!

[joan_massich]

I've open an ticket 'cos it makes no sense. maybe they have some more logs