Custom package with dependency init error: ModuleNotFoundError or ImportError

Question

I'm creating a custom package that has the following structure:

test_package
    │   README.md
    │   setup.py
    │
    ├───my_package
    │       my_package.py
    │       __init__.py
    │
    └───tests
            tests.py

My package depends on pygdbmi, so I added it to the list of required dependencies.

I'm using __init__.py to import the modules

__init__.py :

from .my_package import my_class

__version__ = '0.0.1'
__title__ = 'my_package'

And my class is:

my_package.py :

from pygdbmi.gdbcontroller import GdbController

class my_class:
    def __init__(self):
        print("my_class!!")
        self.gdbmi = GdbController()

The problem is that when I run python setup.py install clean , I get a ModuleNotFoundError :

python setup.py install clean
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    import my_package
  File "c:\test_package\my_package\__init__.py", line 1, in <module>
    from .my_package import my_class
  File "c:\test_package\my_package\my_package.py", line 2, in <module>
    from pygdbmi.gdbcontroller import GdbController
ModuleNotFoundError: No module named 'pygdbmi

Which is obvious, because __init__ is importing my_package , it breaks because I don't have pygdbmi installed yet.

I tried to fix this by removing the import from __init__.py :

__version__ = '0.0.1'
__title__ = 'my_package'

It installs correctly, but now I can't import my package. when I try to run some tests:

python tests.py
Traceback (most recent call last):
  File "tests.py", line 3, in <module>
    from my_package import my_class
ImportError: cannot import name 'my_class' from 'my_package' (C:\Users\lalalala\AppData\Local\Continuum\anaconda3-32\lib\site-packages\my_package-0.0.1-py3.7.egg\my_package\__init__.py)

How do I fix this? I'd like to keep the __init__ structure of defining the version, etc, as it seems the Pythonic way to do it.

Many thanks!!!

setup.py :

import setuptools
import my_package

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    # Project information
    name=my_package.__title__,
    version=my_package.__version__,
    long_description_content_type="text/markdown",
    packages=setuptools.find_packages(),
    install_requires=["pygdbmi"],
    python_requires='>=3.7',
    # Tests
    test_suite='tests'
)

my_package.py :

from pygdbmi.gdbcontroller import GdbController

class my_class:
    def __init__(self):
        print("my_class!!")
        self.gdbmi = GdbController()

tests.py :

import unittest

from my_package import my_class

class some_test(unittest.TestCase):
    def test_constructor(self):
        self.assertIsNotNone(my_class())

if __name__ == '__main__':
    unittest.main()

Answer 1

How do I fix this? I'd like to keep the __init__ structure of defining the version, etc, as it seems the Pythonic way to do it.

It is not, name and version are package metadata, which does not belong into the source code. It belongs into your package definition, meaning setup.py in your case.

Your __init__.py should work the other way round, and get its info from interacting with the python environment and its own installation:

from importlib import metadata

# this works, but usually people just write the name as a string here.
# not 100% DRY, but it's not like the package name could ever change
__title__ = __name__
# if you're stuck on python 3.7 or older, importlib-metadata is a 
# third-party package that can be used as a drop-in instead
__version__ = metadata.version(__title__)

---

The most important take-away message should be that you should never import your code into your build-script. It can 1) create nasty chicken-egg problems where your code can't be built unless an older version of it was installed already, and 2) turn all your run-time dependencies into built-time dependencies, which is what you're experiencing.

You may be able to work around both of these problem, but the easier way would be to set up your package data in the way described above, and never get the issue in the first place.

Answer 2

This is going to sound disappointing but I would pretty much just give up and do the following:

• remove pygdbmi from install_requires
• put pygdbmi in requirements.txt
• wrap the from pygdbmi.gdbcontroller import GdbController in a try...except block which will print a helpful message telling the user they need to install pygdbmi manually if they wish to use the package; then re-raise the exception
• in any installation instructions or deployment scripts just add pip install -r requirements.txt before python setup.py install

As far as I can tell this is not a very uncommon practice - I have met with a couple of packages from PyPI which would require me to install their prerequisites manually.