Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
Level
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

How to update the cache automatically Edited

I'm fairly new to CI and even newer to bitbucket CI.   What I want to do is cache the conda virtual environment that I create, update it incrementally as we add new packages to the environment.yml file, but also be able to newly create since it apparently gets dropped each week.  

My file looks like as follows, and it works to a degree, but with two main issues

image: continuumio/anaconda3

pipelines:
default:
- step:
name: Build
caches:
- condacache
script:
# - conda env create -f environment.yml
- conda activate vlm_venv
- conda env update -f environment.yml
- python tests/tests.py

definitions:
caches:
condacache: /opt/conda/envs/vlm_venv 

 
Issue #1
Note that the first bullet in the "script" section is commented out.  This is due to the fact that the env has already been created and cached, so there's no need to recreate it.  It actually results in this error if that line is uncommented:

CondaValueError: prefix already exists: /opt/conda/envs/vlm_venv

But what happens in 1 week when condacache no longer exists?  How can run that command IFF condacache does not exist?  Or should I not be concerned, because once we're at 1 week, it will replace the dropped cache with the latest build? 

 

Issue #2

As I add packages to environment.yml, the `conda env update` command properly installs them, but since condacache already exists, it doesn't update with the new updated environment containing these new packages.  Rather, the build teardown process notes the following:

You already have a 'condacache' cache so we won't create it again
Cache "condacache": Skipping upload for existing cache

So, the question here is how can I replace the condacache either (1) every time or (2) whenever packages were added?  Either would work, I suppose? 

Thanks in advance.

1 answer

0 votes

@Mike Prince I am not an expert in python development, but I don't think we need to worry about Issue #1 since the environment will be created if it doesn't exist.

Issue #2 is a bit tricky since we need to invalidate the cache when environment.yml is updated. You can use the cache clear pipe to achieve it.

image: continuumio/anaconda3

pipelines:
default:
- step:
name: Clear cache if environments updated
condition:
changesets:
includePaths:
- "environment.yml"
script:
- pipe: atlassian/bitbucket-clear-cache:3.1.1
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
CACHES: ["condacache"]
- step:
name: Build
caches:
- condacache
script:
- conda env create -f environment.yml
- conda activate vlm_venv
- conda env update -f environment.yml
- python tests/tests.py

 There is an open feature request to invalidate the cache when dependencies are updated, which should make it easier to invalidate the cache without a pipe. Please vote for that issue. 

I hope the above steps should resolve your issue, please let me know if you have further questions.

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

Calling any interview participants for Bitbucket Data Center

Hi everyone,  We are looking to learn more about development teams’ workflows and pain points, especially around DevOps, integrations, administration, scale, security, and the related challeng...

496 views 5 4
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you