How to update the cache automatically

Mike Prince April 3, 2021

I'm fairly new to CI and even newer to bitbucket CI.   What I want to do is cache the conda virtual environment that I create, update it incrementally as we add new packages to the environment.yml file, but also be able to newly create since it apparently gets dropped each week.  

My file looks like as follows, and it works to a degree, but with two main issues

image: continuumio/anaconda3

pipelines:
default:
- step:
name: Build
caches:
- condacache
script:
# - conda env create -f environment.yml
- conda activate vlm_venv
- conda env update -f environment.yml
- python tests/tests.py

definitions:
caches:
condacache: /opt/conda/envs/vlm_venv 

 
Issue #1
Note that the first bullet in the "script" section is commented out.  This is due to the fact that the env has already been created and cached, so there's no need to recreate it.  It actually results in this error if that line is uncommented:

CondaValueError: prefix already exists: /opt/conda/envs/vlm_venv

But what happens in 1 week when condacache no longer exists?  How can run that command IFF condacache does not exist?  Or should I not be concerned, because once we're at 1 week, it will replace the dropped cache with the latest build? 

 

Issue #2

As I add packages to environment.yml, the `conda env update` command properly installs them, but since condacache already exists, it doesn't update with the new updated environment containing these new packages.  Rather, the build teardown process notes the following:

You already have a 'condacache' cache so we won't create it again
Cache "condacache": Skipping upload for existing cache

So, the question here is how can I replace the condacache either (1) every time or (2) whenever packages were added?  Either would work, I suppose? 

Thanks in advance.

1 answer

0 votes
Justin Thomas
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
April 5, 2021

@Mike Prince I am not an expert in python development, but I don't think we need to worry about Issue #1 since the environment will be created if it doesn't exist.

Issue #2 is a bit tricky since we need to invalidate the cache when environment.yml is updated. You can use the cache clear pipe to achieve it.

image: continuumio/anaconda3

pipelines:
default:
- step:
name: Clear cache if environments updated
condition:
changesets:
includePaths:
- "environment.yml"
script:
- pipe: atlassian/bitbucket-clear-cache:3.1.1
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
CACHES: ["condacache"]
- step:
name: Build
caches:
- condacache
script:
- conda env create -f environment.yml
- conda activate vlm_venv
- conda env update -f environment.yml
- python tests/tests.py

 There is an open feature request to invalidate the cache when dependencies are updated, which should make it easier to invalidate the cache without a pipe. Please vote for that issue. 

I hope the above steps should resolve your issue, please let me know if you have further questions.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events