Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in

How to configure the cache for the conda package manager in bitbucket pipelines

David Michael Gang
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
February 12, 2018

Hi all,

I use the following pipeline for checking the source code:

 

# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: continuumio/miniconda3:latest

pipelines:
default:
- step:
script: # Modify the commands below to build your repository.
- conda env update -n root --file environment.yml
- pylint module1
- mypy -m module1 --ignore-missing-imports --strict
- coverage run --source=module1 -m unittest discover
- coverage report -m

I want to use the bitbucket caches to make the pipeline faster but i am not sure which directories i should cache.

I already asked the question at the conda user group   but did not get an answer.

How can i configure the cache ?

 

Thanks,

David

3 answers

1 accepted

1 vote
Answer accepted
Rajat Jain May 28, 2018
image: python:3.6.2
pipelines:
default:
- step:
caches:
- condacache
script:
- /opt/python/bin/conda update -y conda
- /opt/python/bin/pip install --upgrade pip
- /opt/python/bin/pip install -r requirements.txt
- /opt/python/bin/pytest

definitions:
caches:
condacache: /opt/python/bin


#This is how i used to create cache in my pipeline: Please take it as reference.
#This is a custom cache technique. Hope it would help you.
tasubo
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
June 6, 2019

This does not work.

0 votes
Denise Skidmore
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
May 22, 2023

You can get the location of your package cache from the command: conda info

Add this to one of your steps and inspect the results, then you can define

your cache as in the other answers.

https://docs.anaconda.com/free/anaconda/packages/shared-pkg-cache/

https://support.atlassian.com/bitbucket-cloud/docs/cache-dependencies/

0 votes
Michiel May 29, 2020

When using the `continuumio/miniconda3` image in your pipeline, I have found that defining a `conda` cache as `/opt/conda/pkgs` worked. After the first run of the pipeline, in subsequent runs of the pipeline the command `conda env update` should not longer print "Downloading and Extracting Packages"

joan_massich
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
June 2, 2020

This does not work for me.

image: continuumio/miniconda3:latest

pipelines:
default:
- step:
caches:
- conda
- codatarballs

script:
- conda env update -n test_env --file environment.yml

definitions:
caches:
conda: ~/.conda/envs/test_env
codatarballs: /opt/conda/pkgs

The output I get is:

Assembling contents of new cache 'conda'
Assembling contents of new cache 'codatarballs'

but then on the next run:

Cache "conda": Downloading
Cache "conda": Not found
Cache "codatarballs": Downloading
Cache "codatarballs": Not found
Michiel June 3, 2020

I don't know why it doesn't download the cache in the next run, it does for me, and I have the same output at the end of the first run:

Assembling contents of new cache 'conda'

One thing you can check is in the main pipelines screen of the repository, there should be a button on the top right named "Caches". If I click that I see:

conda: opt/conda/pkgs 350.31 MB 29 May 2020

Hope this helps!

joan_massich
I'm New Here
I'm New Here
Those new to the Atlassian Community have posted less than three times. Give them a warm welcome!
June 3, 2020

I've open an ticket 'cos it makes no sense. maybe they have some more logs

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events