Can be stored cache shared between step script and after-script?

[etreskovskis]

Why cached pip and venv are not shared with after-script? Is it normal behavior? For example:

definitions:
  caches:
    pip: ~/.cache/pip
    virtualenv: venv

- step:
    name: Run Tests
    caches: [ pip, virtualenv]
    script:
      # script logic
    

    after-script:
      # after script logic

[Theodora Boudale · Accepted solution]

Hi @etreskovskis,

The cache directories and files should be available in the after-script as well.

If you execute the command

ls -lah ~/.cache/pip

in the after-script, do you see the chache's files and directories listed?

If not, can you see them if you execute this command during the script?

Kind regards,
Theodora

[etreskovskis]

I haven't tried.. However, I have a step like

- step:
    name: Install dependencies
    caches: [ pip, virtualenv ]
    script:
      - echo "DISPLAY CURRENT SETUP"
      - which pip
      - which python
      - python --version
      - python -m pip install virtualenv
      - python -m virtualenv venv
      - source venv/bin/activate
      - python -m pip install --upgrade pip
      - python -m pip install -r requirements.txt

which cache my installed requirements.txt
However `after-script` misses these installed/cached requirements

after-script:
  - python -m services.google_api.gdrive_api "$BITBUCKET_BRANCH"

the script fails, that some of the dependencies were not installed.
so workaround was 

after-script:
  - python -m pip install -r requirements.txt
  - python -m services.google_api.gdrive_api "$BITBUCKET_BRANCH"

I perform additionally `pip install -r requirements.txt`
Is it normal behavior? Or cached requirements could be passed in different way without performing pip install -r requirements?

[Theodora Boudale]

Hi @etreskovskis,

Thank you for the info.

I ran a few builds with a script and after-script similar to yours; I used a very minimal requirements.txt file. I also added the command pip list at the end of the script, and one more time at the beginning of the after-script. I saw that packages listed in the pip list command of the script were not available when running pip list in the after-script.

After looking further into it, I believe the command source venv/bin/activate sets or changes some variables. Any changes in variables during the script will not be preserved in the after-script, and this is expected behavior. Pipelines creates a script with all commands of your step's script and runs it. Then, it creates a new script with all the commands of your step's after-script, and runs it in a new session. Therefore, any changes in variables during the script will not be preserved in the after-script.

The issue has nothing to do with cache. With or without cache, any files you create during the script should be available in the after-script.

I suggest trying one of the following:

1. Either use the command

- source venv/bin/activate

in the after-script as well (instead of installing from requirements.txt again).

2. Or remove the commands that create the virtual environment from your script. Every step of a Pipelines build and every step of a different build runs in a separate Docker container, which is destroyed when the step is finished. So, you have a new environment every time you run this step, and you may not need the virtual environment in this case.

Please feel free to let me know how it goes.

Kind regards,
Theodora

[etreskovskis]

Thanks!