Has anyone used GDAL with a python bitbucket pipeline

[Smiler]

I am trying to configure a bitbucket pipeline to unit test our python code. However when I try and use pip to install python if fails. The first problem was that it said that numpy wasn't install despite it being before gdal in the requirements.txt file. So I moved the GDAL install out from the requirements file and tried installed it afterwards. The pipeline definition is

# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: python:3.6.8

clone:
  depth: 1

pipelines:
  custom:
    test:
      - step:
          name: Run unit tests
          caches:
            - pip
          script: # Modify the commands below to build your repository.
            - python -V
            - pip install --upgrade pip
            - pip install -r requirements2.txt
            - pip freeze
            - pip install GDAL
            - python -m unittest discover -s test

The requirements2.txt file contains

numpy==1.17.2
pandas==0.25.1
parse==1.12.1
python-dateutil==2.8.0
pytz==2019.3
six==1.12.0

The output of the commands is

+ python -V
Python 3.6.8

+ pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Found existing installation: pip 19.1.1
Uninstalling pip-19.1.1:
Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.3

+ pip install -r requirements2.txt
Collecting numpy==1.17.2 (from -r requirements2.txt (line 1))
Downloading https://files.pythonhosted.org/packages/e5/e6/c3fdc53aed9fa19d6ff3abf97dfad768ae3afce1b7431f7500000816bda5/numpy-1.17.2-cp36-cp36m-manylinux1_x86_64.whl (20.4MB)
Collecting pandas==0.25.1 (from -r requirements2.txt (line 2))
Downloading https://files.pythonhosted.org/packages/73/9b/52e228545d14f14bb2a1622e225f38463c8726645165e1cb7dde95bfe6d4/pandas-0.25.1-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)
Collecting parse==1.12.1 (from -r requirements2.txt (line 3))
Downloading https://files.pythonhosted.org/packages/84/4d/5578a484f4a3ec5d4440a95731a104a96a12a6fb57385292d3e7b74d8f6d/parse-1.12.1.tar.gz
Collecting python-dateutil==2.8.0 (from -r requirements2.txt (line 4))
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
Collecting pytz==2019.3 (from -r requirements2.txt (line 5))
Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
Collecting six==1.12.0 (from -r requirements2.txt (line 6))
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: parse
Building wheel for parse (setup.py): started
Building wheel for parse (setup.py): finished with status 'done'
Created wheel for parse: filename=parse-1.12.1-cp36-none-any.whl size=22341 sha256=cada607fad94b4bf7661c09817a8d868139f02bfaf2823c5e82a377f2d66d506
Stored in directory: /root/.cache/pip/wheels/16/f1/d5/72f7e2c22ab3d106bb6f7d727552b5e027b832284f4704b106
Successfully built parse
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, parse
Successfully installed numpy-1.17.2 pandas-0.25.1 parse-1.12.1 python-dateutil-2.8.0 pytz-2019.3 six-1.12.0

+ pip freeze
numpy==1.17.2
pandas==0.25.1
parse==1.12.1
python-dateutil==2.8.0
pytz==2019.3
six==1.12.0

+ pip install GDAL==2.4.2
Collecting GDAL==2.4.2
Downloading https://files.pythonhosted.org/packages/dc/d5/90339b48bdcabc76124eaa058a32d796963a05624eb418ed5ea5af7db9fa/GDAL-2.4.2.tar.gz (564kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xnjkin2o/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xnjkin2o/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-xnjkin2o/GDAL/
Complete output (72 lines):
running egg_info
creating pip-egg-info/GDAL.egg-info
writing pip-egg-info/GDAL.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/GDAL.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/GDAL.egg-info/top_level.txt
writing manifest file 'pip-egg-info/GDAL.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 151, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 237, in get_gdal_config
return fetch_config(option, gdal_config=self.gdal_config)
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 154, in fetch_config
raise gdal_config_error(e)
__main__.gdal_config_error: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 151, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 431, in <module>
setup(**setup_kwargs)
File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
File "/usr/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/local/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/local/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 287, in finalize_options
self.gdaldir = self.get_gdal_config('prefix')
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 243, in get_gdal_config
return fetch_config(option)
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 154, in fetch_config
raise gdal_config_error(e)
__main__.gdal_config_error: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Has anyone encountered the same problem and know how to fix it?

Thanks in advance,
Ian

[Daniil Penkin · Accepted solution]

Hello @Smiler, 

Thanks for reaching out.

Pipelines execute the commands of your script within the Docker image you specified, so I believe this particular question is rather about setting up python build in that Docker image properly. Have you tried running your pipeline locally? It is way easier to debug it locally first.

Cheers,
Daniil

[Smiler]

I had been trying to do just that, thanks for the link.

[Smiler]

I solved the problem by creating my own docker image that contained the required version of GDAL and python

[Syed Mohammed Ali Hussaini]

Hi @Smiler  I am trying to do the exact same thing while I do not want to create docker image containing this gdal is it possible you can tell me the process you went through in creating the pipeline with your custom docker image once the image is created and is already present in dockerhub as a private image how do we pull the image (the image is aprox 600 mb )