Help in improving pip usage

Question

I am using pip in three different steps. pip install -r requirements.txt takes almost 60 seconds to run. So I am spending several minutes installing the same requirements in three different spots. What am I missing and how can I make this better? Simplified workflow.. but even with the cache it is killing me image: python:3.6.3 options: docker: true pipelines: branches: master: - parallel: - step: name: Build Images caches: - docker - pip script: - pip install -r requirements.txt - ./build/build.sh - step: name: Test Code caches: - pip script: - pip install -r requirements.txt - python ./manage.py test services: - postgres - step: name: Do something else caches: - pip script: - pip install -r requirements.txt - python ./do_something_else definitions: services: postgres: image: postgres:10.3 Most of the pip time is spent downloading.. can I also cache that somehow? Collecting python-memcached (from djcacheutils==3.0.0->-r requirements.txt (line 18)) Downloading https://files.pythonhosted.org/packages/f5/90/19d3908048f70c120ec66a39e61b92c253e834e6e895cd104ce5e46cbe53/python_memcached-1.59-py2.py3-none-any.whl
Collecting django-appconf (from django-celery-email==2.0.0->-r requirements.txt (line 20)) Downloading https://files.pythonhosted.org/packages/5b/78/726cdf3e04660560cf25f9def95b8f2736310c581dabed9adfe60154a9f8/django_appconf-1.0.2-py2.py3-none-any.whl
Collecting py-moneyed>=0.7 (from django-money==0.12.3->-r requirements.txt (line 21)) Downloading https://files.pythonhosted.org/packages/4c/2c/3c57645cddb0e64aac10749829a843eb5e3aab8e067e2c870d98c38a7947/py_moneyed-0.7.0-py3-none-any.whl
(212kB)Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage==1.8.0->-r requirements.txt (line 29)) Downloading https://files.pythonhosted.org/packages/e3/39/b082c5bd0095d2baf6638338b66c5b6d2e8e372f605a1a85f33c0abe5533/google_api_core-1.1.1-py2.py3-none-any.whl Collecting sqlparse>=0.2.0 (from django-debug-toolbar==1.9.1->-r requirements.txt (line 17)) Downloading https://files.pythonhosted.org/packages/65/85/20bdd72f4537cf2c4d5d005368d502b2f464ede22982e724a82c86268eda/sqlparse-0.2.4-py2.py3-none-any.whl

Brian Wawok_o8jnio · Answer

For future me..

This is what I did and it helped a fair bit

I defined a new cache of the install directory for python3.6

definitions:
caches:
python-packages: /usr/local/lib/python3.6/site-packages/

And then I added this cache in addition to the pip cache to my steps

name: Test Code
caches:
  - pip
  - python-packages
script:
  - pip install -r requirements.txt
  - pip install -r requirements-test.txt
  - python ./manage.py test

That took my total pip down (including downloading caches) down from:

75 seconds (no caches)

60 seconds (pre-defined pip cache)

10 seconds (adding in python-packages cache)