Create
cancel
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Deleted user
0 / 0 points
Next:
badges earned

Your Points Tracker
Challenges
Leaderboard
  • Global
  • Feed

Badge for your thoughts?

You're enrolled in our new beta rewards program. Join our group to get the inside scoop and share your feedback.

Join group
Recognition
Give the gift of kudos
You have 0 kudos available to give
Who do you want to recognize?
Why do you want to recognize them?
Kudos
Great job appreciating your peers!
Check back soon to give more kudos.

Past Kudos Given
No kudos given
You haven't given any kudos yet. Share the love above and you'll see it here.

It's not the same without you

Join the community to find out what other Atlassian users are discussing, debating and creating.

Atlassian Community Hero Image Collage

Has anyone used GDAL with a python bitbucket pipeline

I am trying to configure a bitbucket pipeline to unit test our python code. However when I try and use pip to install python if fails. The first problem was that it said that numpy wasn't install despite it being before gdal in the requirements.txt file. So I moved the GDAL install out from the requirements file and tried installed it afterwards. The pipeline definition is

# This is a sample build configuration for Python.
# Check our guides at https://confluence.atlassian.com/x/x4UWN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: python:3.6.8

clone:
  depth: 1

pipelines:
  custom:
    test:
      - step:
          name: Run unit tests
          caches:
            - pip
          script: # Modify the commands below to build your repository.
            - python -V
            - pip install --upgrade pip
            - pip install -r requirements2.txt
            - pip freeze
            - pip install GDAL
            - python -m unittest discover -s test

The requirements2.txt file contains

numpy==1.17.2
pandas==0.25.1
parse==1.12.1
python-dateutil==2.8.0
pytz==2019.3
six==1.12.0

The output of the commands is

+ python -V
Python 3.6.8

+ pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Found existing installation: pip 19.1.1
Uninstalling pip-19.1.1:
Successfully uninstalled pip-19.1.1
Successfully installed pip-19.2.3

+ pip install -r requirements2.txt
Collecting numpy==1.17.2 (from -r requirements2.txt (line 1))
Downloading https://files.pythonhosted.org/packages/e5/e6/c3fdc53aed9fa19d6ff3abf97dfad768ae3afce1b7431f7500000816bda5/numpy-1.17.2-cp36-cp36m-manylinux1_x86_64.whl (20.4MB)
Collecting pandas==0.25.1 (from -r requirements2.txt (line 2))
Downloading https://files.pythonhosted.org/packages/73/9b/52e228545d14f14bb2a1622e225f38463c8726645165e1cb7dde95bfe6d4/pandas-0.25.1-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)
Collecting parse==1.12.1 (from -r requirements2.txt (line 3))
Downloading https://files.pythonhosted.org/packages/84/4d/5578a484f4a3ec5d4440a95731a104a96a12a6fb57385292d3e7b74d8f6d/parse-1.12.1.tar.gz
Collecting python-dateutil==2.8.0 (from -r requirements2.txt (line 4))
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
Collecting pytz==2019.3 (from -r requirements2.txt (line 5))
Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
Collecting six==1.12.0 (from -r requirements2.txt (line 6))
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Building wheels for collected packages: parse
Building wheel for parse (setup.py): started
Building wheel for parse (setup.py): finished with status 'done'
Created wheel for parse: filename=parse-1.12.1-cp36-none-any.whl size=22341 sha256=cada607fad94b4bf7661c09817a8d868139f02bfaf2823c5e82a377f2d66d506
Stored in directory: /root/.cache/pip/wheels/16/f1/d5/72f7e2c22ab3d106bb6f7d727552b5e027b832284f4704b106
Successfully built parse
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, parse
Successfully installed numpy-1.17.2 pandas-0.25.1 parse-1.12.1 python-dateutil-2.8.0 pytz-2019.3 six-1.12.0

+ pip freeze
numpy==1.17.2
pandas==0.25.1
parse==1.12.1
python-dateutil==2.8.0
pytz==2019.3
six==1.12.0

+ pip install GDAL==2.4.2
Collecting GDAL==2.4.2
Downloading https://files.pythonhosted.org/packages/dc/d5/90339b48bdcabc76124eaa058a32d796963a05624eb418ed5ea5af7db9fa/GDAL-2.4.2.tar.gz (564kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xnjkin2o/GDAL/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xnjkin2o/GDAL/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-xnjkin2o/GDAL/
Complete output (72 lines):
running egg_info
creating pip-egg-info/GDAL.egg-info
writing pip-egg-info/GDAL.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/GDAL.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/GDAL.egg-info/top_level.txt
writing manifest file 'pip-egg-info/GDAL.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 151, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 237, in get_gdal_config
return fetch_config(option, gdal_config=self.gdal_config)
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 154, in fetch_config
raise gdal_config_error(e)
__main__.gdal_config_error: [Errno 2] No such file or directory: '../../apps/gdal-config': '../../apps/gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 151, in fetch_config
p = subprocess.Popen([command, args], stdout=subprocess.PIPE)
File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 431, in <module>
setup(**setup_kwargs)
File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
File "/usr/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/usr/local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/local/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/local/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 287, in finalize_options
self.gdaldir = self.get_gdal_config('prefix')
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 243, in get_gdal_config
return fetch_config(option)
File "/tmp/pip-install-xnjkin2o/GDAL/setup.py", line 154, in fetch_config
raise gdal_config_error(e)
__main__.gdal_config_error: [Errno 2] No such file or directory: 'gdal-config': 'gdal-config'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Has anyone encountered the same problem and know how to fix it?

Thanks in advance,
Ian

 

1 answer

1 accepted

0 votes
Answer accepted

Hello @Smiler

Thanks for reaching out.

Pipelines execute the commands of your script within the Docker image you specified, so I believe this particular question is rather about setting up python build in that Docker image properly. Have you tried running your pipeline locally? It is way easier to debug it locally first.

Cheers,
Daniil

I had been trying to do just that, thanks for the link.

I solved the problem by creating my own docker image that contained the required version of GDAL and python

Like Daniil Penkin likes this

Hi @Smiler  I am trying to do the exact same thing while I do not want to create docker image containing this gdal is it possible you can tell me the process you went through in creating the pipeline with your custom docker image once the image is created and is already present in dockerhub as a private image how do we pull the image (the image is aprox 600 mb ) 

Suggest an answer

Log in or Sign up to answer
TAGS
Community showcase
Published in Bitbucket

New improvements to user management in Bitbucket Cloud 👥

Hey Community! We’re willing to wager that quite a few of you not only use Bitbucket, but administer it too. Our team is excited to share that we’ll be releasing improvements throughout this month of...

518 views 4 13
Read article

Community Events

Connect with like-minded Atlassian users at free events near you!

Find an event

Connect with like-minded Atlassian users at free events near you!

Unfortunately there are no Community Events near you at the moment.

Host an event

You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events

Events near you