IPython on Windows fails to import geopandas

Question

I get an

ImportError: DLL load failed: The specified procedure could not be found.

error when trying to import geopandas in python 3.6. Specifically, I get the error when using ipython but not when using python. Also, this affects Windows (a Windows Server 2016 virtual machine) and not Linux. I've found a few previous posts on this, or very similar issues, but I'm rejecting their suitability as they either don't clearly resolve the problem or else conflate it with pip installs.

This pos t from nearly two years ago, for example, reports a similar error, but concludes with a “Never mind, I did a pip install of geopandas”.

This post from just over a couple of years ago has an accepted answer despite the original poster commenting that it didn't work for them! There's a mention of a blog post from Geoff Boeing that I've seen before as providing a working method, despite that blog post providing more than one approach (a conda install and a more manual sequence of steps) and the comment not clarifying what worked for them.

There's this post from nearly two and a half years ago that conflates conda and pip install methods and doesn't have an accepted answer. There's a suggestion in a comment that, for the commenter, it was an issue with gdal on conda-forge. There's an answer that refers to Geoff Boeing's blogpost again. The implication may be that the install of gdal via conda can be problematic and, if it is, then the manual sequence of steps is required. I am not persuaded this is my issue.

My problem occurs specifically on a Windows Server 2016 virtual machine and when specifying only the conda-forge channel. Also, pertinently, it only occurs in ipython (and thus Jupyter notebooks) and not in python, thus:

Create environment specifying the conda defaults channel, specifying python 3.6, ipython, and geopandas:

conda create -n test36_defaults -c defaults python=3.6 ipython geopandas

Activate that environment:

(test36_2) C:\Users\guy.maskall>conda activate test36_defaults

Fire up an ipython kernel and import geopandas:

(test36_defaults) C:\Users\guy.maskall>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import geopandas as gpd

In [2]:
Do you really want to exit ([y]/n)?

It imported okay.

Similarly, fire up the straight python interpreter:

(test36_defaults) C:\Users\guy.maskall>python
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import geopandas as gpd
>>>

That imported okay as well.

Now create another environment specifying conda-forge only:

conda create -n test36_cforge -c conda-forge python=3.6 ipython geopandas

Activate the environment and load ipython again to import geopandas:

(test36_cforge) C:\Users\guy.maskall>ipython
Python 3.6.11 (default, Aug  5 2020, 19:41:03) [MSC v.1916 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import geopandas as gpd
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-a62d01c1d62e> in <module>()
----> 1 import geopandas as gpd

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\geopandas\__init__.py in <module>()
      5 from geopandas.array import points_from_xy  # noqa
      6
----> 7 from geopandas.io.file import _read_file as read_file  # noqa
      8 from geopandas.io.arrow import _read_parquet as read_parquet  # noqa
      9 from geopandas.io.arrow import _read_feather as read_feather  # noqa

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\geopandas\io\file.py in <module>()
      5 import pandas as pd
      6
----> 7 import fiona
      8 import pyproj
      9 from shapely.geometry import mapping

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__init__.py in <module>()
     84 import fiona._loading
     85 with fiona._loading.add_gdal_dll_directories():
---> 86     from fiona.collection import BytesCollection, Collection
     87     from fiona.drvsupport import supported_drivers
     88     from fiona.env import ensure_env_with_credentials, Env

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\collection.py in <module>()
      9 with fiona._loading.add_gdal_dll_directories():
     10     from fiona import compat, vfs
---> 11     from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
     12     from fiona.ogrext import Session, WritingSession
     13     from fiona.ogrext import buffer_to_virtual_file, remove_virtual_file, GEOMETRY_TYPES

ImportError: DLL load failed: The specified procedure could not be found.

In [2]:
Do you really want to exit ([y]/n)?

So that failed in ipython. Now try it using just straight python:

(test36_cforge) C:\Users\guy.maskall>python
Python 3.6.11 (default, Aug  5 2020, 19:41:03) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import geopandas as gpd
>>>

That works.

One of the answers in the above SO posts highlights GDAL, Fiona, pyproj, rtree, and shapely. Comparing the output of conda list between the two environments above, I get exactly the same versions of them all. All but rtree even have the same build. So I have:

gdal                      3.1.4            py36h214b664_0    conda-forge  
fiona                     1.8.17           py36hdef4c2b_1 conda-forge  
pyproj                    2.6.1.post1      py36hfcef96e_3 conda-forge  
shapely                   1.7.1            py36h6dc46f0_1 conda-forge

and

rtree                     0.9.4            py36h9a6d676_1    conda-forge

(associated with the env that didn't work) vs

rtree                     0.9.4            py36h21ff451_1

(in the env that does work).

What's quite confusing me is that despite some posts implicating gdal, and my error output above implicating fiona loading gdal (I think), both environments seem to have exactly the same version and build of gdal and fiona, and the error only occurs in the ipython kernel and not in python.

It's important for me to be able to easily replicate environments, and so conda is preferred. I have requirements other than simply getting geopandas to load. For example, I use datacube, which recommends python 3.6. I was previously running a higher version of python and I'd like to get a python 3.6 environment up and running so I can see if that resolves another issue I was having. I'd like to narrow down what is causing this error as I want to be able to easily edit my conda env yaml to try different environments, and I may need to specify multiple channels as one suspect (for another issue) is prompt-toolkit, which only seems available in a version below 3 from esrf-bcu channel, but that's another story.

So can anyone help point to

• What, exactly, is the package/version/build that could be causing this?
• Why on earth, even in the problem environment, it only impacts ipython?

There's a small difference in python version (3.6.11 failing with 3.6.12 succeeding), and that's something I can explore, but that still leaves my question of why the difference between ipython vs python.

In other news, I hate Windows.

Importing fiona and DLL path

Update 2020/11/02 As the error seems to centre on fiona, I've boiled it down to just that. In the anaconda prompt shell, I ran set PYTHONVERBOSE=1 , then loaded ipython and tried to import fiona:

In [1]: import fiona
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\__init__.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__init__.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\__init__.cpython-36.pyc'
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\_loading.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\_loading.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\_loading.cpython-36.pyc'
import 'fiona._loading' # <_frozen_importlib_external.SourceFileLoader object at 0x0000020344DC0C88>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\__pycache__\socket.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\socket.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\__pycache__\\socket.cpython-36.pyc'
# extension module '_socket' loaded from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\DLLs\\_socket.pyd'
# extension module '_socket' executed from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\DLLs\\_socket.pyd'
import '_socket' # <_frozen_importlib_external.ExtensionFileLoader object at 0x0000020344DCED30>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\__pycache__\selectors.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\selectors.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\__pycache__\\selectors.cpython-36.pyc'
import 'selectors' # <_frozen_importlib_external.SourceFileLoader object at 0x0000020344DD9358>
import 'socket' # <_frozen_importlib_external.SourceFileLoader object at 0x0000020344DCE198>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\collection.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\collection.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\collection.cpython-36.pyc'
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\compat.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\compat.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\compat.cpython-36.pyc'
import 'fiona.compat' # <_frozen_importlib_external.SourceFileLoader object at 0x0000020344DE76D8>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\vfs.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\vfs.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\vfs.cpython-36.pyc'
import 'fiona.vfs' # <_frozen_importlib_external.SourceFileLoader object at 0x0000020344DE7780>
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-4c762272c079> in <module>()
----> 1 import fiona

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _load_unlocked(spec)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap_external.py in exec_module(self, module)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__init__.py in <module>()
     84 import fiona._loading
     85 with fiona._loading.add_gdal_dll_directories():
---> 86     from fiona.collection import BytesCollection, Collection
     87     from fiona.drvsupport import supported_drivers
     88     from fiona.env import ensure_env_with_credentials, Env

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _load_unlocked(spec)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap_external.py in exec_module(self, module)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\collection.py in <module>()
      9 with fiona._loading.add_gdal_dll_directories():
     10     from fiona import compat, vfs
---> 11     from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
     12     from fiona.ogrext import Session, WritingSession
     13     from fiona.ogrext import buffer_to_virtual_file, remove_virtual_file, GEOMETRY_TYPES

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _load_unlocked(spec)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in module_from_spec(spec)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap_external.py in create_module(self, spec)

C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

ImportError: DLL load failed: The specified procedure could not be found.

But then exiting ipython and firing up straight python and trying to import fiona, I get a tonne more output (saving you all of it, but focusing on the bit where ipython seems to bail out):

...

import 'fiona.compat' # <_frozen_importlib_external.SourceFileLoader object at 0x00000289AC978470>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\vfs.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\vfs.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\vfs.cpython-36.pyc'
import 'fiona.vfs' # <_frozen_importlib_external.SourceFileLoader object at 0x00000289AC997550>
# extension module 'fiona.ogrext' loaded from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\ogrext.cp36-win_amd64.pyd'
# extension module 'fiona._geometry' loaded from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\_geometry.cp36-win_amd64.pyd'
# extension module 'fiona._err' loaded from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\_err.cp36-win_amd64.pyd'
# extension module 'fiona._err' executed from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\_err.cp36-win_amd64.pyd'
import 'fiona._err' # <_frozen_importlib_external.ExtensionFileLoader object at 0x00000289AC9EE7F0>
# C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\__pycache__\errors.cpython-36.pyc matches C:\Users\guy.maskall\anaconda3\envs\test36_cforge\lib\site-packages\fiona\errors.py
# code object from 'C:\\Users\\guy.maskall\\anaconda3\\envs\\test36_cforge\\lib\\site-packages\\fiona\\__pycache__\\errors.cpython-36.pyc'
import 'fiona.errors' # <_frozen_importlib_external.SourceFileLoader object at 0x00000289AC9EEFD0>

...

This is diving into guts I'm really not familiar with! But it seems I can safely say a few things:

• Both ipython and python are importing from the same path (as you'd hope)
• After the point at which ipython bails, python reports/comments about extension module fiona.ogrext being loaded, which is the last error reported in the traceback
• Furthermore, that error is within the context of fiona._loading.add_gdal_dll_directories(), suggesting it's something to do with DLL paths

But again I'm back to needing to ask why this function, or finding the DLLs, behaves differently in ipython as compared to python?

Also, I've verified I see this same behaviour on Windows 10 Pro Version 2004.

Answer 1

The specific import issue has been resolved simply by specifying python 3.6.12 from 3.6.11, thus:

conda create -n test36_12_cforge -c conda-forge python=3.6.12 ipython geopandas
(test36_12_cforge) C:\Users\guy.maskall>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import geopandas as gpd

In [2]:
Do you really want to exit ([y]/n)?

(test36_12_cforge) C:\Users\guy.maskall>ipython
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:29:25) [MSC v.1916 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import fiona

In [2]:
Do you really want to exit ([y]/n)?

I'm reluctant to accept my own answer. For a beer if we ever meet, can anyone explain why there'd be this difference in behaviour depending on whether the import is being done from ipython vs python?