How to resolve Pyinstaller module not found error

Question

I am trying to use Pyinstaller to package my project into an executable. Currently, I am doing it on my Ubuntu based PC for proof of concept, after which I plan to switch to Windows to build an.exe which can be run on windows. The issue is that after I build the executable and run it, it cannot find one of the local packages which were imported in the startup file. This is my project structure:

.
├── hook-streamlit.py
├── __init__.py
├── models
│   ├── __init__.py
│   ├── loader.py
│   ├── nn.py
│   └── runner.py
├── notebooks
├── README.md
├── requirements.txt
├── stapp
│   ├── __init__.py
│   ├── main.py
│   └── session_state.py
├── startup.py
├── startup.spec
├── test_data
│   ├── test_x.csv
│   ├── test_y.csv
│   ├── train_x.csv
│   └── train_y.csv
├── tkapp.py
├── unipredictor-icon.ico
├── UniPredictor.spec
└── utils.py

startup.py is the startup script and these are the contents:

import os
import subprocess
import shlex

from models import nn, loader, runner
from stapp import main

subprocess.call(shlex.split(f"streamlit run {os.path.join('stapp', 'main.py')} --global.developmentMode=false"))

Even though I don't need the models and stapp packages in the startup script, I import them, just to ensure that pyinstaller resolves the dependencies since I use subprocess to run the app. But even with this, I still get ModuleNotFoundError: No module named 'models' after building and running the executable. This error comes from stapp.main where models is imported. I have tried adding both models and stapp to hiddenimports without success. I would think that since the project root is added to PYTHONPATH, it shouldn't have any issue with local packages. And even if for some reason it did, the import and hiddenimports should deal with that. Here is my current spec file:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['startup.py'],
             pathex=['/home/kenneth/PycharmProjects/universal_predictor'],
             binaries=[],
             datas=[],
             hiddenimports=['models', 'stapp'],
             hookspath=['.'],
             runtime_hooks=[],
             excludes=['torch.distributions'],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='startup',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          upx_exclude=[],
          runtime_tmpdir=None,
          console=False , icon='unipredictor-icon.ico')

I will appreciate any help on how to make pyinstaller include the project's local packages. Thanks!

Answer 1

I resolved the issue by loading the needed packages and modules as data.

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['startup.py'],
             pathex=['/home/kenneth/PycharmProjects/universal_predictor'],
             binaries=[],
              datas=[('.streamlit', '.streamlit'), ('.data', '.data'), ('models', 'models'), ('stapp', 'stapp'), ('utils.py', '.')],
             hiddenimports=[],
             hookspath=['.'],
             runtime_hooks=[],
             excludes=['torch.distributions'],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='startup',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          upx_exclude=[],
          runtime_tmpdir=None,
          console=False , icon='unipredictor-icon.ico')

Answer 2

The problem is exactly what is being shown. You need to extend the PYTHONPATH for models. In your spec file add the package path for models as well.

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['startup.py'],
             pathex=['/home/kenneth/PycharmProjects/universal_predictor', 
'/home/kenneth/PycharmProjects/universal_predictor/models'],
             binaries=[],
             datas=[],
             hiddenimports=['models', 'stapp'],
             hookspath=['.'],
             runtime_hooks=[],
             excludes=['torch.distributions'],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='startup',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          upx_exclude=[],
          runtime_tmpdir=None,
          console=False , icon='unipredictor-icon.ico')