Where does pip install packages in pipelines?

[justinimoser]

Hi,

I have just started using pipelines, and I have a yml file as follows:

image: python:3.6.4

clone: 
    depth: full

pipelines: 
    default: 
        - step: name: Installing dependancies 
            caches: 
                - pip 
            script: 
                - pip install -r requirements.txt -vvv 
        - step:  
            name: Debugging dependancy installs and locations 
            script: 
                - ls -altr /usr/local/lib/python3.6/site-packages 
                - ls -altr /usr/local/bin 
                - which pip 
                - which python 
                - python -c "import site; print(site.getsitepackages())" 
        - step: 
            name: Installing local app script: 
                - python setup.py install 
                - ls -altr /usr/local/lib/python3.6/site-packages 
        - step: 
            name: Setting environment variables 
            script: 
                - export FLASK_APP=stackademicdata 
        - step: 
            name: Run Flask app 
            caches: 
                - flask 
            script: 
                - python -m flask run 
        - step: 
            deployment: test 
            name: Deploy to Heroku 
            script: 
                - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_DATA_APP_NAME.git HEADdefinitions: 
caches: 
    flask: /usr/local/bin/flask

the contents of my requirement.txt is:

flask
flask-sqlalchemy
click
gunicorn
pytest
setuptools
jsonify
requests
entrypoints
decorators
psycopg2
sqlalchemy
jsonschema
flask-wtf

the output states that all packages in this step have been installed correctly:

Successfully installed 

Jinja2-2.10 
MarkupSafe-1.0 
WTForms-2.1 
Werkzeug-0.14.1 
attrs-17.4.0 
certifi-2018.1.18 
chardet-3.0.4 
click-6.7 
decorators-0.1.1 
entrypoints-0.2.3 
flask-0.12.2 flask-sqlalchemy-2.3.2 
flask-wtf-0.14.2 
gunicorn-19.7.1 
idna-2.6 
itsdangerous-0.24 
jsonify-0.5 
jsonschema-2.6.0 
pluggy-0.6.0 
psycopg2-2.7.4 
py-1.5.2 
pytest-3.4.2 
requests-2.18.4 
six-1.11.0 
sqlalchemy-1.2.5 
urllib3-1.22

Cleaning up..

Therefore I would expect to be able to use those packages in the standard way, with CLI control. For example, with the flask package, in my build file I need to be able to run 'flask run' to test whether the application starts correctly.

However, the build always fails on that step, with the error:

+ python -m flask run/usr/local/bin/python: No module named flask

I have included debugging steps, but for the life of me I cant figure out where pip installs the packages?!

This is the output of the debugging step commands:

+ ls -altr /usr/local/lib/python3.6/site-packages
total 96
-rw-r--r--. 1 root staff 119 Mar 14 17:49 README.txt
drwxr-sr-x. 1 root staff 4096 Mar 14 17:49 ..
drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 wheel-0.30.0.dist-info
drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 wheel
drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 setuptools-38.5.2.dist-info
drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 setuptools
drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 pkg_resources
drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 pip-9.0.1.dist-info
drwxr-sr-x. 11 root staff 4096 Mar 14 17:49 pip
-rw-r--r--. 1 root staff 126 Mar 14 17:49 easy_install.py
drwxr-sr-x. 1



+ ls -altr /usr/local/bin
total 188
drwxrwsr-x. 1 root staff 4096 Mar 12 00:00 ..
-rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6m
-rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6
-rwxr-xr-x. 1 root staff 441 Mar 14 17:49 pyvenv-3.6
lrwxrwxrwx. 1 root staff 10 Mar 14 17:49 pyvenv -> pyvenv-3.6
-rwxr-xr-x. 1 root staff 3109 Mar 14 17:49 python3.6m-config
lrwxrwxrwx. 1 root staff 17 Mar 14 17:49 python3.6-config -> python3.6m-config
lrwxrwxrwx. 1 root staff 16 Mar 14 17:49 python3-config -> python3.6-config
lrwxrwxrwx. 1 root staff 9 Mar 14 17:49 python3 -> python3.6
-rwxr-xr-x. 1 root staff 84 Mar 14 17:49 pydoc3.6
lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 pydoc3 -> pydoc3.6
-rwxr-xr-x. 1 root staff 99 Mar 14 17:49 idle3.6
lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 idle3 -> idle3.6
-rwxr-xr-x. 1 root staff 101 Mar 14 17:49 2to3-3.6
lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 2to3 -> 2to3-3.6
lrwxrwxrwx. 1 root staff 14 Mar 14 17:49 python-config -> python3-config
lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 python -> python3
lrwxrwxrwx. 1 root staff 6 Mar 14 17:49 pydoc -> pydoc3
lrwxrwxrwx. 1 root staff 5 Mar 14 17:49 idle -> idle3
-rwxr-xr-x. 1 root staff 218 Mar 14 17:49 wheel
-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3.6
-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3
-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip
-rwxr-xr-x. 1 root staff 239 Mar 14 17:49 easy_install-3.6
-rwxr-xr-x. 1 r


+ which pip
/usr/local/bin/pip

+ which python
/usr/local/bin/python

But I cannot find the flask package (or any others for that matter!)

Please help!

Thanks,

Justin

[Jeroen De Raedt]

Since you are using python 3, shouldn't you use

pip3 install -r requirements.txt -vvv 

[justinimoser]

I assumed I would have to use pip as that is the cached dependancy in pipelines? It doesnt mention pip3....

[justinimoser]

I  updated the pip steps to use pip3 and it made no difference.

What I find most puzzling, is that at the end of the first step (pip3 install...)

this is the output:

changing mode of /usr/local/bin/flask to 755

So it definitely is there but, as you can see, Ive added that as a custom cache at the bottom, but the build still fails with the same error.

+ python -m flask run/usr/local/bin/python: No module named flask

[Barend Scholtus]

I read each step creates a new docker image. Could it be related?

I can't get it to work either. I've got a pretty simple:

pipelines:
default:
- step:
name: Build
caches:
- pip
- setuptools

.......

definitions:
caches:
- setuptools: setuptools ~/.cache/setuptools

But at the end of each build, I always get:

Cache "setuptools": Skipping upload for empty cache

[ank]

Did we get any solution.