Hi, I have just started using pipelines, and I have a yml file as follows: image: python:3.6.4 clone: depth: full pipelines: default: - step: name: Installing dependancies caches: - pip script: - pip install -r requirements.txt -vvv - step: name: Debugging dependancy installs and locations script: - ls -altr /usr/local/lib/python3.6/site-packages - ls -altr /usr/local/bin - which pip - which python - python -c "import site; print(site.getsitepackages())" - step: name: Installing local app script: - python setup.py install - ls -altr /usr/local/lib/python3.6/site-packages - step: name: Setting environment variables script: - export FLASK_APP=stackademicdata - step: name: Run Flask app caches: - flask script: - python -m flask run - step: deployment: test name: Deploy to Heroku script: - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_DATA_APP_NAME.git HEADdefinitions: caches: flask: /usr/local/bin/flask the contents of my requirement.txt is: flask flask-sqlalchemy click gunicorn pytest setuptools jsonify requests entrypoints decorators psycopg2 sqlalchemy jsonschema flask-wtf the output states that all packages in this step have been installed correctly: Successfully installed Jinja2-2.10 MarkupSafe-1.0 WTForms-2.1 Werkzeug-0.14.1 attrs-17.4.0 certifi-2018.1.18 chardet-3.0.4 click-6.7 decorators-0.1.1 entrypoints-0.2.3 flask-0.12.2 flask-sqlalchemy-2.3.2 flask-wtf-0.14.2 gunicorn-19.7.1 idna-2.6 itsdangerous-0.24 jsonify-0.5 jsonschema-2.6.0 pluggy-0.6.0 psycopg2-2.7.4 py-1.5.2 pytest-3.4.2 requests-2.18.4 six-1.11.0 sqlalchemy-1.2.5 urllib3-1.22 Cleaning up.. Therefore I would expect to be able to use those packages in the standard way, with CLI control. For example, with the flask package, in my build file I need to be able to run 'flask run' to test whether the application starts correctly. However, the build always fails on that step, with the error: + python -m flask run/usr/local/bin/python: No module named flask I have included debugging steps, but for the life of me I cant figure out where pip installs the packages?! This is the output of the debugging step commands: + ls -altr /usr/local/lib/python3.6/site-packages total 96 -rw-r--r--. 1 root staff 119 Mar 14 17:49 README.txt drwxr-sr-x. 1 root staff 4096 Mar 14 17:49 .. drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 wheel-0.30.0.dist-info drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 wheel drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 setuptools-38.5.2.dist-info drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 setuptools drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 pkg_resources drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 pip-9.0.1.dist-info drwxr-sr-x. 11 root staff 4096 Mar 14 17:49 pip -rw-r--r--. 1 root staff 126 Mar 14 17:49 easy_install.py drwxr-sr-x. 1 + ls -altr /usr/local/bin total 188 drwxrwsr-x. 1 root staff 4096 Mar 12 00:00 .. -rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6m -rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6 -rwxr-xr-x. 1 root staff 441 Mar 14 17:49 pyvenv-3.6 lrwxrwxrwx. 1 root staff 10 Mar 14 17:49 pyvenv -> pyvenv-3.6 -rwxr-xr-x. 1 root staff 3109 Mar 14 17:49 python3.6m-config lrwxrwxrwx. 1 root staff 17 Mar 14 17:49 python3.6-config -> python3.6m-config lrwxrwxrwx. 1 root staff 16 Mar 14 17:49 python3-config -> python3.6-config lrwxrwxrwx. 1 root staff 9 Mar 14 17:49 python3 -> python3.6 -rwxr-xr-x. 1 root staff 84 Mar 14 17:49 pydoc3.6 lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 pydoc3 -> pydoc3.6 -rwxr-xr-x. 1 root staff 99 Mar 14 17:49 idle3.6 lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 idle3 -> idle3.6 -rwxr-xr-x. 1 root staff 101 Mar 14 17:49 2to3-3.6 lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 2to3 -> 2to3-3.6 lrwxrwxrwx. 1 root staff 14 Mar 14 17:49 python-config -> python3-config lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 python -> python3 lrwxrwxrwx. 1 root staff 6 Mar 14 17:49 pydoc -> pydoc3 lrwxrwxrwx. 1 root staff 5 Mar 14 17:49 idle -> idle3 -rwxr-xr-x. 1 root staff 218 Mar 14 17:49 wheel -rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3.6 -rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3 -rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip -rwxr-xr-x. 1 root staff 239 Mar 14 17:49 easy_install-3.6 -rwxr-xr-x. 1 r + which pip /usr/local/bin/pip + which python /usr/local/bin/python But I cannot find the flask package (or any others for that matter!) Please help! Thanks, Justin

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Where does pip install packages in pipelines?

justinimoser

Hi,

I have just started using pipelines, and I have a yml file as follows:

image: python:3.6.4

clone: 
    depth: full

pipelines: 
    default: 
        - step: name: Installing dependancies 
            caches: 
                - pip 
            script: 
                - pip install -r requirements.txt -vvv 
        - step:  
            name: Debugging dependancy installs and locations 
            script: 
                - ls -altr /usr/local/lib/python3.6/site-packages 
                - ls -altr /usr/local/bin 
                - which pip 
                - which python 
                - python -c "import site; print(site.getsitepackages())" 
        - step: 
            name: Installing local app script: 
                - python setup.py install 
                - ls -altr /usr/local/lib/python3.6/site-packages 
        - step: 
            name: Setting environment variables 
            script: 
                - export FLASK_APP=stackademicdata 
        - step: 
            name: Run Flask app 
            caches: 
                - flask 
            script: 
                - python -m flask run 
        - step: 
            deployment: test 
            name: Deploy to Heroku 
            script: 
                - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_DATA_APP_NAME.git HEADdefinitions: 
caches: 
    flask: /usr/local/bin/flask

the contents of my requirement.txt is:

flask
flask-sqlalchemy
click
gunicorn
pytest
setuptools
jsonify
requests
entrypoints
decorators
psycopg2
sqlalchemy
jsonschema
flask-wtf

the output states that all packages in this step have been installed correctly:

<span>Cleaning up..</span><span>Successfully installed <br><br>Jinja2-2.10 <br>MarkupSafe-1.0 <br>WTForms-2.1 <br>Werkzeug-0.14.1 <br>attrs-17.4.0 <br>certifi-2018.1.18 <br>chardet-3.0.4 <br>click-6.7 <br>decorators-0.1.1 <br>entrypoints-0.2.3 <br>flask-0.12.2 flask-sqlalchemy-2.3.2 <br>flask-wtf-0.14.2 <br>gunicorn-19.7.1 <br>idna-2.6 <br>itsdangerous-0.24 <br>jsonify-0.5 <br>jsonschema-2.6.0 <br>pluggy-0.6.0 <br>psycopg2-2.7.4 <br>py-1.5.2 <br>pytest-3.4.2 <br>requests-2.18.4 <br>six-1.11.0 <br>sqlalchemy-1.2.5 <br>urllib3-1.22<br><br></span>

Therefore I would expect to be able to use those packages in the standard way, with CLI control. For example, with the flask package, in my build file I need to be able to run 'flask run' to test whether the application starts correctly.

However, the build always fails on that step, with the error:

+ python -m flask run/usr/local/bin/python: No module named flask

I have included debugging steps, but for the life of me I cant figure out where pip installs the packages?!

This is the output of the debugging step commands:

<span>+ ls -altr /usr/local/lib/python3.6/site-packages<br>total 96<br>-rw-r--r--. 1 root staff 119 Mar 14 17:49 README.txt<br>drwxr-sr-x. 1 root staff 4096 Mar 14 17:49 ..<br>drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 wheel-0.30.0.dist-info<br>drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 wheel<br>drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 setuptools-38.5.2.dist-info<br>drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 setuptools<br>drwxr-sr-x. 5 root staff 4096 Mar 14 17:49 pkg_resources<br>drwxr-sr-x. 2 root staff 4096 Mar 14 17:49 pip-9.0.1.dist-info<br>drwxr-sr-x. 11 root staff 4096 Mar 14 17:49 pip<br>-rw-r--r--. 1 root staff 126 Mar 14 17:49 easy_install.py<br>drwxr-sr-x. 1<br><br><br><br>+ ls -altr /usr/local/bin<br>total 188<br>drwxrwsr-x. 1 root staff 4096 Mar 12 00:00 ..<br>-rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6m<br>-rwxr-xr-x. 2 root staff 14312 Mar 14 17:49 python3.6<br>-rwxr-xr-x. 1 root staff 441 Mar 14 17:49 pyvenv-3.6<br>lrwxrwxrwx. 1 root staff 10 Mar 14 17:49 pyvenv -> pyvenv-3.6<br>-rwxr-xr-x. 1 root staff 3109 Mar 14 17:49 python3.6m-config<br>lrwxrwxrwx. 1 root staff 17 Mar 14 17:49 python3.6-config -> python3.6m-config<br>lrwxrwxrwx. 1 root staff 16 Mar 14 17:49 python3-config -> python3.6-config<br>lrwxrwxrwx. 1 root staff 9 Mar 14 17:49 python3 -> python3.6<br>-rwxr-xr-x. 1 root staff 84 Mar 14 17:49 pydoc3.6<br>lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 pydoc3 -> pydoc3.6<br>-rwxr-xr-x. 1 root staff 99 Mar 14 17:49 idle3.6<br>lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 idle3 -> idle3.6<br>-rwxr-xr-x. 1 root staff 101 Mar 14 17:49 2to3-3.6<br>lrwxrwxrwx. 1 root staff 8 Mar 14 17:49 2to3 -> 2to3-3.6<br>lrwxrwxrwx. 1 root staff 14 Mar 14 17:49 python-config -> python3-config<br>lrwxrwxrwx. 1 root staff 7 Mar 14 17:49 python -> python3<br>lrwxrwxrwx. 1 root staff 6 Mar 14 17:49 pydoc -> pydoc3<br>lrwxrwxrwx. 1 root staff 5 Mar 14 17:49 idle -> idle3<br>-rwxr-xr-x. 1 root staff 218 Mar 14 17:49 wheel<br>-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3.6<br>-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip3<br>-rwxr-xr-x. 1 root staff 211 Mar 14 17:49 pip<br>-rwxr-xr-x. 1 root staff 239 Mar 14 17:49 easy_install-3.6<br>-rwxr-xr-x. 1 r<br><br><br>+ which pip<br>/usr/local/bin/pip<br><br>+ which python<br>/usr/local/bin/python<br></span>

But I cannot find the flask package (or any others for that matter!)

Please help!

Thanks,

Justin

Accepted answers

All comments

Jeroen De Raedt

Since you are using python 3, shouldn't you use

pip3 install -r requirements.txt -vvv

justinimoser

I assumed I would have to use pip as that is the cached dependancy in pipelines? It doesnt mention pip3....

justinimoser

I updated the pip steps to use pip3 and it made no difference.

What I find most puzzling, is that at the end of the first step (pip3 install...)

this is the output:

changing mode of /usr/local/bin/flask to 755

So it definitely is there but, as you can see, Ive added that as a custom cache at the bottom, but the build still fails with the same error.

+ python -m flask run/usr/local/bin/python: No module named flask

Barend Scholtus

I read each step creates a new docker image. Could it be related?

I can't get it to work either. I've got a pretty simple:

pipelines:
default:
- step:
name: Build
caches:
- pip
- setuptools

.......

definitions:
caches:
- setuptools: setuptools ~/.cache/setuptools

But at the end of each build, I always get:

Cache "setuptools": Skipping upload for empty cache

Welcome!

It looks like you're new here. Sign in or register to get started.

Quick Links

Find more posts tagged with

bitbucket

bitbucket-pipelines