How to update the cache automatically

Question

I'm fairly new to CI and even newer to bitbucket CI. What I want to do is cache the conda virtual environment that I create, update it incrementally as we add new packages to the environment.yml file, but also be able to newly create since it apparently gets dropped each week. My file looks like as follows, and it works to a degree, but with two main issues image: continuumio/anaconda3 pipelines: default: - step: name: Build caches: - condacache script: # - conda env create -f environment.yml - conda activate vlm_venv - conda env update -f environment.yml - python tests/tests.py definitions: caches: condacache: /opt/conda/envs/vlm_venv Issue #1 Note that the first bullet in the "script" section is commented out. This is due to the fact that the env has already been created and cached, so there's no need to recreate it. It actually results in this error if that line is uncommented: CondaValueError: prefix already exists: /opt/conda/envs/vlm_venv But what happens in 1 week when condacache no longer exists? How can run that command IFF condacache does not exist? Or should I not be concerned, because once we're at 1 week, it will replace the dropped cache with the latest build? Issue #2 As I add packages to environment.yml, the `conda env update` command properly installs them, but since condacache already exists, it doesn't update with the new updated environment containing these new packages. Rather, the build teardown process notes the following: You already have a 'condacache' cache so we won't create it again
Cache "condacache": Skipping upload for existing cache So, the question here is how can I replace the condacache either (1) every time or (2) whenever packages were added? Either would work, I suppose? Thanks in advance.

Justin Thomas · Answer

@Mike Prince I am not an expert in python development, but I don't think we need to worry about Issue #1 since the environment will be created if it doesn't exist.

Issue #2 is a bit tricky since we need to invalidate the cache when environment.yml is updated. You can use the cache clear pipe to achieve it.

image: continuumio/anaconda3

pipelines:
default:
    - step:
        name: Clear cache if environments updated
         condition:
             changesets:
                 includePaths:
                   - "environment.yml"  
        script:
          - pipe: atlassian/bitbucket-clear-cache:3.1.1
             variables:
                 BITBUCKET_USERNAME: $BITBUCKET_USERNAME
                 BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
                 CACHES: ["condacache"]
    - step: 
name: Build
caches:
          - condacache
script:
          - conda env create -f environment.yml 
          - conda activate vlm_venv
          - conda env update -f environment.yml
          - python tests/tests.py

There is an open feature request to invalidate the cache when dependencies are updated, which should make it easier to invalidate the cache without a pipe. Please vote for that issue.

I hope the above steps should resolve your issue, please let me know if you have further questions.