Some specific Python packages are not installed by PIP via requirements.txt

[Danny Chia]

I've set up a pipeline to run some Python scripts. Here's my requirements.txt file that lists the dependencies:

matplotlib
numpy
openpyxl
pandas
PyYAML
tabulate
influxdb-client
wiliot-api
xlrd
xlwt
prettytable
scipy
kaleido==0.2.1
plotly==5.14.1

I use the following commands to install the packages:

steps:
  - step: &setup
      name: 'Install dependencies'
      caches:
        - pip3
      script:
        - python3 -m pip install --upgrade pip
        - pip install -r requirements.txt

However, two packages, specifically Plotly and kaleodo, are not installed when pip is first run. I have to run pip specifically for those packages to install them. Here's an excerpt from my bitbucket-pipelines.yml file:

- pip list -v
- pip install kaleido
- pip install plotly
- pip list -v
- python foo.py

The first invocation of "pip list -v" does not show those packages:

Package Version Location Installer
------------------ -------- --------------------------------------- ---------
certifi 2023.5.7 /usr/local/lib/python3.11/site-packages pip
charset-normalizer 3.1.0 /usr/local/lib/python3.11/site-packages pip
contourpy 1.0.7 /usr/local/lib/python3.11/site-packages pip
cycler 0.11.0 /usr/local/lib/python3.11/site-packages pip
et-xmlfile 1.1.0 /usr/local/lib/python3.11/site-packages pip
fonttools 4.39.4 /usr/local/lib/python3.11/site-packages pip
graphql-core 3.2.3 /usr/local/lib/python3.11/site-packages pip
idna 3.4 /usr/local/lib/python3.11/site-packages pip
influxdb-client 1.36.1 /usr/local/lib/python3.11/site-packages pip
kiwisolver 1.4.4 /usr/local/lib/python3.11/site-packages pip
matplotlib 3.7.1 /usr/local/lib/python3.11/site-packages pip
numpy 1.24.3 /usr/local/lib/python3.11/site-packages pip
openpyxl 3.1.2 /usr/local/lib/python3.11/site-packages pip
packaging 23.1 /usr/local/lib/python3.11/site-packages pip
pandas 2.0.2 /usr/local/lib/python3.11/site-packages pip
Pillow 9.5.0 /usr/local/lib/python3.11/site-packages pip
pip 23.1.2 /usr/local/lib/python3.11/site-packages pip
prettytable 3.7.0 /usr/local/lib/python3.11/site-packages pip
PyJWT 2.7.0 /usr/local/lib/python3.11/site-packages pip
pyparsing 3.0.9 /usr/local/lib/python3.11/site-packages pip
python-dateutil 2.8.2 /usr/local/lib/python3.11/site-packages pip
pytz 2023.3 /usr/local/lib/python3.11/site-packages pip
PyYAML 6.0 /usr/local/lib/python3.11/site-packages pip
reactivex 4.0.4 /usr/local/lib/python3.11/site-packages pip
requests 2.31.0 /usr/local/lib/python3.11/site-packages pip
scipy 1.10.1 /usr/local/lib/python3.11/site-packages pip
seaborn 0.12.2 /usr/local/lib/python3.11/site-packages pip
setuptools 65.5.0 /usr/local/lib/python3.11/site-packages pip
setuptools-scm 7.1.0 /usr/local/lib/python3.11/site-packages pip
sgqlc 16.1 /usr/local/lib/python3.11/site-packages pip
six 1.16.0 /usr/local/lib/python3.11/site-packages pip
tabulate 0.9.0 /usr/local/lib/python3.11/site-packages pip
typing_extensions 4.6.2 /usr/local/lib/python3.11/site-packages pip
tzdata 2023.3 /usr/local/lib/python3.11/site-packages pip
urllib3 2.0.2 /usr/local/lib/python3.11/site-packages pip
wcwidth 0.2.6 /usr/local/lib/python3.11/site-packages pip
wheel 0.38.4 /usr/local/lib/python3.11/site-packages pip
wiliot-api 4.1.2 /usr/local/lib/python3.11/site-packages pip
xlrd 2.0.1 /usr/local/lib/python3.11/site-packages pip
xlwt 1.3.0 /usr/local/lib/python3.11/site-packages pip

 Only after I run pip again for those two packages, do they show up:

Package Version Location Installer
------------------ -------- --------------------------------------- ---------
certifi 2023.5.7 /usr/local/lib/python3.11/site-packages pip
charset-normalizer 3.1.0 /usr/local/lib/python3.11/site-packages pip
contourpy 1.0.7 /usr/local/lib/python3.11/site-packages pip
cycler 0.11.0 /usr/local/lib/python3.11/site-packages pip
et-xmlfile 1.1.0 /usr/local/lib/python3.11/site-packages pip
fonttools 4.39.4 /usr/local/lib/python3.11/site-packages pip
graphql-core 3.2.3 /usr/local/lib/python3.11/site-packages pip
idna 3.4 /usr/local/lib/python3.11/site-packages pip
influxdb-client 1.36.1 /usr/local/lib/python3.11/site-packages pip
kaleido 0.2.1 /usr/local/lib/python3.11/site-packages pip
kiwisolver 1.4.4 /usr/local/lib/python3.11/site-packages pip
matplotlib 3.7.1 /usr/local/lib/python3.11/site-packages pip
numpy 1.24.3 /usr/local/lib/python3.11/site-packages pip
openpyxl 3.1.2 /usr/local/lib/python3.11/site-packages pip
packaging 23.1 /usr/local/lib/python3.11/site-packages pip
pandas 2.0.2 /usr/local/lib/python3.11/site-packages pip
Pillow 9.5.0 /usr/local/lib/python3.11/site-packages pip
pip 23.1.2 /usr/local/lib/python3.11/site-packages pip
plotly 5.14.1 /usr/local/lib/python3.11/site-packages pip
prettytable 3.7.0 /usr/local/lib/python3.11/site-packages pip
PyJWT 2.7.0 /usr/local/lib/python3.11/site-packages pip
pyparsing 3.0.9 /usr/local/lib/python3.11/site-packages pip
python-dateutil 2.8.2 /usr/local/lib/python3.11/site-packages pip
pytz 2023.3 /usr/local/lib/python3.11/site-packages pip
PyYAML 6.0 /usr/local/lib/python3.11/site-packages pip
reactivex 4.0.4 /usr/local/lib/python3.11/site-packages pip
requests 2.31.0 /usr/local/lib/python3.11/site-packages pip
scipy 1.10.1 /usr/local/lib/python3.11/site-packages pip
seaborn 0.12.2 /usr/local/lib/python3.11/site-packages pip
setuptools 65.5.0 /usr/local/lib/python3.11/site-packages pip
setuptools-scm 7.1.0 /usr/local/lib/python3.11/site-packages pip
sgqlc 16.1 /usr/local/lib/python3.11/site-packages pip
six 1.16.0 /usr/local/lib/python3.11/site-packages pip
tabulate 0.9.0 /usr/local/lib/python3.11/site-packages pip
tenacity 8.2.2 /usr/local/lib/python3.11/site-packages pip
typing_extensions 4.6.2 /usr/local/lib/python3.11/site-packages pip
tzdata 2023.3 /usr/local/lib/python3.11/site-packages pip
urllib3 2.0.2 /usr/local/lib/python3.11/site-packages pip
wcwidth 0.2.6 /usr/local/lib/python3.11/site-packages pip
wheel 0.38.4 /usr/local/lib/python3.11/site-packages pip
wiliot-api 4.1.2 /usr/local/lib/python3.11/site-packages pip
xlrd 2.0.1 /usr/local/lib/python3.11/site-packages pip
xlwt 1.3.0 /usr/local/lib/python3.11/site-packages pip

 Anyone have an idea why this happens?

[Erez Maadani]

Hi @Danny Chia 

It's a long shot, but try changing your script to:

      script:
        - pip install -r requirements.txt 

Sometimes, the upgrade affects the permissions/locations used by pip and it might lead to weird errors. 

[Danny Chia]

This didn't make a difference for me, but I appreciate the help nonetheless!