Why cached pip and venv are not shared with after-script? Is it normal behavior? For example:
definitions:
caches:
pip: ~/.cache/pip
virtualenv: venv
- step:
name: Run Tests
caches: [ pip, virtualenv]
script:
# script logic
after-script:
# after script logic
Hi @etreskovskis,
The cache directories and files should be available in the after-script as well.
If you execute the command
ls -lah ~/.cache/pip
in the after-script, do you see the chache's files and directories listed?
If not, can you see them if you execute this command during the script?
Kind regards,
Theodora
I haven't tried.. However, I have a step like
- step:
name: Install dependencies
caches: [ pip, virtualenv ]
script:
- echo "DISPLAY CURRENT SETUP"
- which pip
- which python
- python --version
- python -m pip install virtualenv
- python -m virtualenv venv
- source venv/bin/activate
- python -m pip install --upgrade pip
- python -m pip install -r requirements.txt
which cache my installed requirements.txt
However `after-script` misses these installed/cached requirements
after-script:
- python -m services.google_api.gdrive_api "$BITBUCKET_BRANCH"
the script fails, that some of the dependencies were not installed.
so workaround was
after-script:
- python -m pip install -r requirements.txt
- python -m services.google_api.gdrive_api "$BITBUCKET_BRANCH"
I perform additionally `pip install -r requirements.txt`
Is it normal behavior? Or cached requirements could be passed in different way without performing pip install -r requirements?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi @etreskovskis,
Thank you for the info.
I ran a few builds with a script and after-script similar to yours; I used a very minimal requirements.txt file. I also added the command pip list at the end of the script, and one more time at the beginning of the after-script. I saw that packages listed in the pip list command of the script were not available when running pip list in the after-script.
After looking further into it, I believe the command source venv/bin/activate sets or changes some variables. Any changes in variables during the script will not be preserved in the after-script, and this is expected behavior. Pipelines creates a script with all commands of your step's script and runs it. Then, it creates a new script with all the commands of your step's after-script, and runs it in a new session. Therefore, any changes in variables during the script will not be preserved in the after-script.
The issue has nothing to do with cache. With or without cache, any files you create during the script should be available in the after-script.
I suggest trying one of the following:
1. Either use the command
- source venv/bin/activate
in the after-script as well (instead of installing from requirements.txt again).
2. Or remove the commands that create the virtual environment from your script. Every step of a Pipelines build and every step of a different build runs in a separate Docker container, which is destroyed when the step is finished. So, you have a new environment every time you run this step, and you may not need the virtual environment in this case.
Please feel free to let me know how it goes.
Kind regards,
Theodora
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You are very welcome! Please feel free to reach out if you ever need anything else!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.