ModuleNotFoundError when packaging a Python project

Question

I've developed a Python-based simulator and want to package it for the use of public.

Here is my setup.py :

import os

import setuptools
from setuptools import setup

from mypackage.version import __name__, __version__, __author__, __email__, __license__, __url__, \
    __description__, __keywords__, __classifiers__, __python_requires__
# ---------------------------------------------------------------------------------------------------------
# GENERAL
# ---------------------------------------------------------------------------------------------------------

thelibFolder = os.path.dirname(os.path.realpath(__file__))
requirementPath = thelibFolder + '/requirements.txt'
install_requires = []
if os.path.isfile(requirementPath):
    with open(requirementPath) as f:
        install_requires = f.read().splitlines()

data = dict(
    name=__name__,
    version=__version__,
    author=__author__,
    url=__url__,
    python_requires=__python_requires__,
    author_email=__email__,
    description=__description__,
    license=__license__,
    keywords=__keywords__,
    install_requires=install_requires,
    platforms='any',
    classifiers=__classifiers__
    # py_modules=['ipython']
)


# ---------------------------------------------------------------------------------------------------------
# OTHER METADATA
# ---------------------------------------------------------------------------------------------------------


# update the readme.rst to be part of setup
def readme():
    with open('README.md') as f:
        return f.read()


def packages():
    return ["mypackage"] + ["mypackage." + e for e in setuptools.find_packages(where='mypackage')]


data['long_description'] = readme()
data['long_description_content_type'] = 'text/x-rst'
data['packages'] = packages()
data['include_package_data'] = True


# ============================================================
# SETUP
# ============================================================


def run_setup(setup_args):
    # retrieve the original input arguments and execute default setup
    kwargs = setup_args
    setup(**kwargs)

    print('*' * 74)
    print("Python installation succeeded.")
    print('*' * 74)


run_setup(data)

And here is my requirements.txt :

ipython==7.29.0
ipython-genutils==0.2.0
attrs==21.2.0
cachetools==4.2.4
certifi==2021.10.8
chardet==4.0.0
cloudpickle==2.0.0
cycler==0.11.0
decorator==5.1.0
future==0.18.2
google-auth==2.3.3
gym==0.21.0
idna==3.3
iniconfig==1.1.1
kiwisolver==1.3.2
kubernetes==19.15.0
matplotlib==3.4.3
networkx==2.6.3
numpy==1.21.4
oauthlib==3.1.1
overloading==0.5.0
packaging==21.2
pandas==1.3.4
Pillow==8.4.0
pluggy==1.0.0
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyglet==1.5.21
pyparsing==3.0.6
pytest==6.2.5
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
scipy==1.7.2
seaborn==0.11.2
simpy==4.0.1
six==1.16.0
sortedcontainers==2.4.0
toml==0.10.2
typing==3.7.4.3
urllib3==1.26.7
websocket-client==1.2.1
prettytable==2.4.0

Then, I run the following to make the package: python3 setup.py sdist

It creates the package for me under dist , and then I can successfully upload it to a hosted repository (in my case gemfury.com). Of course, I can then pull it in a local project on my machine and can use my own package.

Then I try to create another python project, to basically use this package:

In requirements.txt :

--extra-index-url https://<my-token>@pypi.fury.io/<my-username>/
mypackage==0.1.0

In main.py :

from mypackage import MyServer

MyServer.run()

MyServer has a from IPython.display import Image, display in the beginning.

Then I make a Dockerfile to containerize this project:

FROM python:3.10
ADD main.py /
ADD requirements.txt /
RUN pip install -r requirements.txt
CMD [ "python", "./main.py"]

I then run the Dockerfile, it successfully pull mypackage, but then I get the following error:

 ... from IPython.display import Image, display ModuleNotFoundError: No module named 'IPython'

I still get the same error even if I manually add ipython to requirements.txt file in my dockerized project (in addition to having it requirements.txt in mypackage),

Any ideas?

Update 1: Here is the output of my docker build (when I just have mypackage in requirements):

> Preparing build context archive...
> [==================================================>]1153/1153 files
> Done
> 
> Sending build context to Docker daemon...
> [==================================================>] 3.788MB Done
> 
> Step 1/5 : FROM python:3.10  ---> 4246fb19839f 
> Step 2/5 : ADD main.py
> /  ---> Using cache  ---> 8dab230b6201 
> Step 3/5 : ADD requirements.txt
> /  ---> Using cache  ---> 2c56cb36982b 
> Step 4/5 : RUN pip install -r
> requirements.txt  ---> Running in e5ec7c02aaea Looking in indexes:
> https://pypi.org/simple, https://****@pypi.fury.io/michelgokan/
> Collecting mypackage==0.1.0   Downloading
> https://pypi.fury.io/michelgokan/-/ver_SqO4f/mypackage-0.1.0.tar.gz
> (62 kB)
>     ERROR: Command errored out with exit status 1:
>      command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] =
> '"'"'/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/setup.py'"'"';
> __file__='"'"'/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/setup.py'"'"';f
> = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import
> setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
> egg_info --egg-base /tmp/pip-pip-egg-info-fz0vejus
>          cwd: /tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/
>     Complete output (9 lines):
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/setup.py",
> line 6, in <module>
>         from mypackage.version import __name__, __version__, __author__, __email__, __license__, __url__, \
>       File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/mypackage/__init__.py",
> line 16, in <module>
>         from .helpers.utils import Utils
>       File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/mypackage/helpers/utils.py",
> line 11, in <module>
>         from IPython.display import Image, display
>     ModuleNotFoundError: No module named 'IPython'
>     ---------------------------------------- WARNING: Discarding https://pypi.fury.io/username/-/<somthing>/mypackage-0.1.0.tar.gz#sha256=<some-token>
> (from https://pypi.fury.io/michelgokan/mypackage/)
> (requires-python:>=3.9). Command errored out with exit status 1:
> python setup.py egg_info Check the logs for full command output.
> ERROR: Could not find a version that satisfies the requirement
> mypackage==0.1.0 (from versions: 0.1.0)

and this is when I have copy paste the contents of mypackage `requirements.txt` to this new dockerized project's requirements.txt:

> Preparing build context archive...
> [==================================================>]1153/1153 files
> Done
> 
> Sending build context to Docker daemon...
> [==================================================>] 3.788MB Done
> 
> Step 1/5 : FROM python:3.10  ---> 4246fb19839f Step 2/5 : ADD main.py
> /  ---> Using cache  ---> 8dab230b6201 Step 3/5 : ADD requirements.txt
> /  ---> Using cache  ---> 050eebba6079 Step 4/5 : RUN pip install -r
> requirements.txt  ---> Running in edd3d9d868a1 Looking in indexes:
> https://pypi.org/simple, https://****@pypi.fury.io/michelgokan/
> Collecting IPython==7.29.0   Downloading
> ipython-7.29.0-py3-none-any.whl (790 kB) Collecting
> ipython-genutils==0.2.0   Downloading
> ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB) Collecting
> attrs==21.2.0   Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
> Collecting cachetools==4.2.4   Downloading
> cachetools-4.2.4-py3-none-any.whl (10 kB) Collecting
> certifi==2021.10.8   Downloading
> certifi-2021.10.8-py2.py3-none-any.whl (149 kB) Collecting
> chardet==4.0.0   Downloading chardet-4.0.0-py2.py3-none-any.whl (178
> kB) Collecting cloudpickle==2.0.0   Downloading
> cloudpickle-2.0.0-py3-none-any.whl (25 kB) Collecting cycler==0.11.0  
> Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB) Collecting
> decorator==5.1.0   Downloading decorator-5.1.0-py3-none-any.whl (9.1
> kB) Collecting future==0.18.2   Downloading future-0.18.2.tar.gz (829
> kB) Collecting google-auth==2.3.3   Downloading
> google_auth-2.3.3-py2.py3-none-any.whl (155 kB) Collecting gym==0.21.0
> Downloading gym-0.21.0.tar.gz (1.5 MB) Collecting idna==3.3  
> Downloading idna-3.3-py3-none-any.whl (61 kB) Collecting
> iniconfig==1.1.1   Downloading iniconfig-1.1.1-py2.py3-none-any.whl
> (5.0 kB) Collecting kiwisolver==1.3.2   Downloading
> kiwisolver-1.3.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
> (1.6 MB) Collecting kubernetes==19.15.0   Downloading
> kubernetes-19.15.0-py2.py3-none-any.whl (1.7 MB) Collecting
> matplotlib==3.4.3   Downloading matplotlib-3.4.3.tar.gz (37.9 MB)
> Collecting networkx==2.6.3   Downloading
> networkx-2.6.3-py3-none-any.whl (1.9 MB) Collecting numpy==1.21.4  
> Using cached
> numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (15.9 MB) Collecting oauthlib==3.1.1   Downloading
> oauthlib-3.1.1-py2.py3-none-any.whl (146 kB) Collecting
> overloading==0.5.0   Downloading overloading-0.5.0-py3-none-any.whl
> (10 kB) Collecting packaging==21.2   Downloading
> packaging-21.2-py3-none-any.whl (40 kB) Collecting pandas==1.3.4  
> Downloading
> pandas-1.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (11.5 MB) Collecting Pillow==8.4.0   Downloading
> Pillow-8.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (3.1 MB) Collecting pluggy==1.0.0   Downloading
> pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting py==1.11.0  
> Downloading py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting
> pyasn1==0.4.8   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
> Collecting pyasn1-modules==0.2.8   Downloading
> pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting
> pyglet==1.5.21   Downloading pyglet-1.5.21-py3-none-any.whl (1.1 MB)
> Collecting pyparsing==3.0.6   Downloading
> pyparsing-3.0.6-py3-none-any.whl (97 kB) Collecting pytest==6.2.5  
> Downloading pytest-6.2.5-py3-none-any.whl (280 kB) Collecting
> python-dateutil==2.8.2   Downloading
> python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting
> pytz==2021.3   Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)
> Collecting PyYAML==6.0   Downloading
> PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
> (682 kB) Collecting requests==2.26.0   Downloading
> requests-2.26.0-py2.py3-none-any.whl (62 kB) Collecting
> requests-oauthlib==1.3.0   Downloading
> requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB) Collecting
> rsa==4.7.2   Downloading rsa-4.7.2-py3-none-any.whl (34 kB) Collecting
> scipy==1.7.2   Downloading
> scipy-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (39.9 MB) Collecting seaborn==0.11.2   Downloading
> seaborn-0.11.2-py3-none-any.whl (292 kB) Collecting simpy==4.0.1  
> Downloading simpy-4.0.1-py2.py3-none-any.whl (29 kB) Collecting
> six==1.16.0   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
> Collecting sortedcontainers==2.4.0   Downloading
> sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) Collecting
> toml==0.10.2   Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
> Collecting typing==3.7.4.3   Downloading typing-3.7.4.3.tar.gz (78 kB)
> Collecting urllib3==1.26.7   Downloading
> urllib3-1.26.7-py2.py3-none-any.whl (138 kB) Collecting
> websocket-client==1.2.1   Downloading
> websocket_client-1.2.1-py2.py3-none-any.whl (52 kB) Collecting
> prettytable==2.4.0   Downloading prettytable-2.4.0-py3-none-any.whl
> (24 kB) Collecting mypackage==0.1.0   Downloading
> https://pypi.fury.io/michelgokan/-/ver_SqO4f/mypackage-0.1.0.tar.gz (62
> kB)
>     ERROR: Command errored out with exit status 1:
>      command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] =
> '"'"'/tmp/pip-install-0c2n4ub7/mypackage_<some_number>/setup.py'"'"';
> __file__='"'"'/tmp/pip-install-0c2n4ub7/mypackage_<some_number>/setup.py'"'"';f
> = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import
> setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
> '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
> egg_info --egg-base /tmp/pip-pip-egg-info-sogtdamf
>          cwd: /tmp/pip-install-0c2n4ub7/mypackage_<some_number>/
>     Complete output (9 lines):
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File "/tmp/pip-install-0c2n4ub7/mypackage_<some_number>/setup.py",
> line 6, in <module>
>         from mypackage.version import __name__, __version__, __author__, __email__, __license__, __url__, \
>       File "/tmp/pip-install-0c2n4ub7/mypackage_<some_number>/mypackage/__init__.py",
> line 16, in <module>
>         from .helpers.utils import Utils
>       File "/tmp/pip-install-0c2n4ub7/mypackage_<some_number>/mypackage/helpers/utils.py",
> line 11, in <module>
>         from IPython.display import Image, display
>     ModuleNotFoundError: No module named 'IPython'

Answer 1

Based on

Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/setup.py",
line 6, in <module>
        from mypackage.version import __name__, __version__, __author__, __email__, __license__, __url__, \
      File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/mypackage/__init__.py",
line 16, in <module>
        from .helpers.utils import Utils
      File "/tmp/pip-install-z_ofzy_s/mypackage_<some_number_here>/mypackage/helpers/utils.py",
line 11, in <module>
        from IPython.display import Image, display
    ModuleNotFoundError: No module named 'IPython'

your problem is that since you import mypackage.version , mypackage gets imported, which executes __init__.py . __init__.py imports mypackage.helpers.utils , which in turn requires IPython .

In other words, your package's setup.py won't work without IPython installed beforehand, and this is not quite related to Docker – any system without IPython would fail to install your package from source.

You might want to consider using the new declarative setup metadata system instead of a setup.py that needs to import your package, or to build wheels that don't have a setup.py.