pyinstaller: Determine hidden imports for socket.io and aiohttp in Windows Service

Question

I've been using python to build a Windows service that runs socket.io and aiohttp. The program runs perfectly in the interpreter, but once I use pyinstaller to package it into an exe file, the code fails with the exception:

Traceback (most recent call last):
  File "autocoreserver.py", line 24, in __init__
  File "site-packages\socketio\asyncio_server.py", line 67, in __init__
  File "site-packages\socketio\server.py", line 89, in __init__
  File "site-packages\engineio\server.py", line 123, in __init__
ValueError: Invalid async_mode specified

In my program, the exception being thrown is when it tries to create the AsyncServer (File "autocoreserver.py", line 24, in init ).

try:
    self.sio = socketio.AsyncServer()
except:
    rootLogger.exception("AsyncServer init")

I've researched online and know this is a common problem typically solved using the hiddenimports argument in the spec file. However, after adding all the recommendations I could find online, there is no change to the error.

To try and find the modules missing, I built the exe with the "-v" option. However, there are dozens of modules listed, almost none of them having to do with socket.io or aiohttp. For instance, there are many Qt modules missing, but this is a service with no user interface and I'm not using Qt.

Tying to pinpoint an actual list of modules I needed, I instead added this code to my program, which makes a list of all modules in use and writes them out to a file.

    f = open('c:\\modulereport.log','w')
    f.write('hiddenimports=[\n')

    mods = [m.__name__ for m in sys.modules.values() if m]

    for m in mods:
        f.write( '    \'{0}\',\n'.format(m) )

    f.write("],")

    f.close()

This gave me a list of what was supposed to be every module in use by the program, but it was missing several. When I added back all the modules I knew to be missing, I was back to the same error.

I'm new to pyinstaller and python, and I'm guessing there is a better way pinpoint missing modules. Can anyone please tell me the best way to find the modules I'm missing?

This is my spec file (build.spec):

# -*- mode: python -*-

block_cipher = None

#options = [ ('v', None, 'OPTION') ]
options = []

a = Analysis(['autocoreservice.py'],
             pathex=['C:\\Users\\ThomasBitskyJr\\Documents\\srcroot\\3182TeeTable\\AutoCore-Server'],
             binaries=[],
             datas=[],
             hiddenimports=[
                'win32timezone',
                'engineio.async_gevent',
                'gevent',
                'gevent-websocket',
                'aiohttp'
             ],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          options,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='autocoreservice',
          debug=False,
          strip=False,
          upx=True,
          runtime_tmpdir=None,
          console=True )

I build the exe with the following command:

pyinstaller --onefile build.spec

Version Information:

python --version
Python 3.6.5 :: Anaconda, Inc.
pip --version
pip 18.0 from c:\users\thomasbitskyjr\anaconda3\lib\site-packages\pip (python 3.6)
pyinstaller --version
3.3.1

Thanks in advance for any advice.

Update: I improved the situation by updating all the modules on the system. That moved the error to later in the process. Now I get:

Traceback (most recent call last):
  File "C:\Users\User\Documents\srcroot\autocoreserver.py", line 30, in __init__
    self.sio.attach(self.app)
  File "C:\Users\User\Anaconda3\lib\site-packages\socketio\asyncio_server.py", line 74, in attach
    self.eio.attach(app, socketio_path)
  File "C:\Users\User\Anaconda3\lib\site-packages\engineio\asyncio_server.py", line 64, in attach
    self._async['create_route'](app, self, '/{}/'.format(engineio_path))
KeyError: 'create_route'

I've tried both pyinstaller and cx_Freeze and get the same results.

Answer 1

I do not have any good trouble-shooting tips, but I was able to get this exact error resolved. My research mainly showed the issue was due to hidden-imports, but finding what was hidden was a guess and check. Because it's engineio where the error happens, I did some snooping in the site-packages to see what the package directory setup looked like, and came up with the following command, with the flags to include the missing packages

pyinstaller <your_script>.py --hidden-import engineio.async_drivers.aiohttp --hidden-import engineio.async_aiohttp  --distpath some/dist/dir