'No module named util' error when using pyinstaller

Question

I am having trouble running a compiled a script using pyinstaller. I get as far as compiling and then run into the well documented import module problem. I was manually adding these missing modules to the < hiddenimports= > field of the .spec file based on the exceptions they throw during the runtime of the compiled exe. I have unfortunately run into a hitch where it is now asking for 'util'. This is not particularly helpful since this could be a submodule of many different modules.

So far I have added in 'ctypes.util', 'pandas.util', 'distutils.util', which are all of the .util modules contained in the warning file that pyinstaller outputs.

Does anyone know of a common module that could be the cause of this, or if adding them into the .spec file like this is an insufficient way of remedying this problem?

So far to debug I have:

• Included the verbose option [('v',None,'OPTION')], although this doesn't seem to return which modules are missing.

• Checked the warning file which gives some indication of missing modules. I have added in all the '.util' modules that I could see.

The script does use some geospatial modules such as GDAL and Mapnik which bind to external programs and are a possible source of the missing modules.

Warning file was too big to post, so I've located it here.

Feeling slightly defeated and worried that I will never be able to package up this script into an exe. If anyone can advise on how to bug test, or can advise on some of the errors. Much appreciated.

Copy of the .spec file

# -*- mode: python -*-
a = Analysis(['GIS_Helper.py'],
             pathex=['C:\\Users\\Hp\\PycharmProjects\\GISdev'],
             hiddenimports=['scipy.linalg.cython_blas', 'scipy.linalg.cython_lapack', 'scipy.special._ufuncs_cxx', 'multiprocessing', 'multiprocessing.process', 'ctypes.util', 'pandas.util', 'distutils.util'],
             hookspath=None,
             runtime_hooks=None)
pyz = PYZ(a.pure)
exe = EXE(pyz,
          a.scripts,
          [('v',None,'OPTION')],
          a.binaries,
          a.zipfiles,
          a.datas,
          name='GIS_Helper.exe',
          debug=False,
          strip=None,
          upx=True,
          console=True )

Edit: Here is a capture of the runtime Debug using the -verbose flag.

# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
import marshal # builtin
import _struct # builtin
import nt # builtin
import imp # builtin
import zlib # builtin
import errno # builtin
import _weakref # builtin
import _codecs # builtin
import _sre # builtin
import _locale # builtin
import _ctypes # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\_ctypes.pyd
import PIL._imaging # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\PIL._imaging.pyd
import _collections # builtin
import operator # builtin
import itertools # builtin
import _heapq # builtin
import thread # builtin
import _io # builtin
import array # builtin
import math # builtin
import strop # builtin
import _functools # builtin
import binascii # builtin
import _hashlib # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\_hashlib.pyd
import _random # builtin
import cStringIO # builtin
import time # builtin
import datetime # builtin
import _socket # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\_socket.pyd
import _ssl # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\_ssl.pyd
import parser # builtin
import win32api # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\win32api.pyd
import pywintypes # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pywintypes27.dll
import pythoncom # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pythoncom27.dll
import pyproj._proj # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pyproj._proj.pyd
import numpy.core.multiarray # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.core.multiarray.pyd
import numpy.core.umath # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.core.umath.pyd
import cPickle # builtin
import numpy.core._dotblas # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.core._dotblas.pyd
import numpy.core.scalarmath # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.core.scalarmath.pyd
import numpy.lib._compiled_base # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.lib._compiled_base.pyd
import numpy.linalg.lapack_lite # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.linalg.lapack_lite.pyd
import numpy.linalg._umath_linalg # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.linalg._umath_linalg.pyd
import future_builtins # builtin
import numpy.fft.fftpack_lite # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.fft.fftpack_lite.pyd
import numpy.random.mtrand # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\numpy.random.mtrand.pyd
import pandas.hashtable # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.hashtable.pyd
import _winreg # builtin
import gc # builtin
import msvcrt # builtin
import _subprocess # builtin
import _bisect # builtin
import pandas.tslib # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.tslib.pyd
import pandas.lib # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.lib.pyd
import _csv # builtin
import pandas.algos # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.algos.pyd
import pandas.index # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.index.pyd
import pandas._period # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas._period.pyd
import pandas._sparse # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas._sparse.pyd
import matplotlib._path # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\matplotlib._path.pyd
import _ast # builtin
import pandas.parser # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.parser.pyd
import pandas.json # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.json.pyd
import pandas.msgpack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas.msgpack.pyd
import pandas._testing # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\pandas._testing.pyd
import scipy.linalg._fblas # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg._fblas.pyd
import scipy.linalg._flapack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg._flapack.pyd
import scipy.linalg._flinalg # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg._flinalg.pyd
import scipy.linalg._solve_toeplitz # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg._solve_toeplitz.pyd
import scipy.linalg.cython_blas # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg.cython_blas.pyd
import scipy.linalg.cython_lapack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg.cython_lapack.pyd
import scipy.linalg._decomp_update # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.linalg._decomp_update.pyd
import scipy.special._ufuncs_cxx # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.special._ufuncs_cxx.pyd
import scipy.special._ufuncs # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.special._ufuncs.pyd
import scipy.special.specfun # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.special.specfun.pyd
import scipy.integrate._odepack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.integrate._odepack.pyd
import scipy.integrate._quadpack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.integrate._quadpack.pyd
import scipy.integrate.vode # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.integrate.vode.pyd
import scipy.integrate._dop # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.integrate._dop.pyd
import scipy.integrate.lsoda # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.integrate.lsoda.pyd
import scipy.special._ellip_harm_2 # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.special._ellip_harm_2.pyd
import scipy.interpolate._fitpack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.interpolate._fitpack.pyd
import scipy.interpolate.dfitpack # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.interpolate.dfitpack.pyd
import scipy.interpolate._ppoly # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.interpolate._ppoly.pyd
import scipy.sparse._sparsetools # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse._sparsetools.pyd
import scipy.sparse._csparsetools # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse._csparsetools.pyd
import scipy.sparse.csgraph._tools # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse.csgraph._tools.pyd
import scipy.sparse.csgraph._shortest_path # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse.csgraph._shortest_path.pyd
import scipy.sparse.csgraph._traversal # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse.csgraph._traversal.pyd
import scipy.sparse.csgraph._min_spanning_tree # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse.csgraph._min_spanning_tree.pyd
import scipy.sparse.csgraph._reordering # dynamically loaded from C:\Users\Hp\AppData\Local\Temp\_MEI81202\scipy.sparse.csgraph._reordering.pyd
Traceback (most recent call last):
  File "<string>", line 13, in <module>
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\Hp\PycharmProjects\GISdev\build\gis_helper\out00-PYZ.pyz\scipy.interpolate", line 145, in <module>
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\Hp\PycharmProjects\GISdev\build\gis_helper\out00-PYZ.pyz\scipy.interpolate.interpolate", line 31, in <module>
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 409, in load_module
    module = imp.load_module(fullname, fp, filename, self._c_ext_tuple)
  File "scipy/interpolate/interpnd.pyx", line 1, in init scipy.interpolate.interpnd (scipy\interpolate\interpnd.c:24330)
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\Hp\PycharmProjects\GISdev\build\gis_helper\out00-PYZ.pyz\scipy.spatial", line 91, in <module>
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 409, in load_module
    module = imp.load_module(fullname, fp, filename, self._c_ext_tuple)
  File "ckdtree.pyx", line 18, in init ckdtree (scipy\spatial\ckdtree\ckdtree.cxx:22601)
  File "C:\Python27\Lib\site-packages\PyInstaller\loader\pyi_importers.py", line 270, in load_module
    exec(bytecode, module.__dict__)
  File "C:\Users\Hp\PycharmProjects\GISdev\build\gis_helper\out00-PYZ.pyz\multiprocessing", line 65, in <module>
ImportError: No module named util
WARNING: file already exists but should not: C:\Users\Hp\AppData\Local\Temp\_MEI81202\Include\pyconfig.h

Also, output of the build compile

46 INFO: Testing for ability to set icons, version resources...
77 INFO: ... resource update available
77 INFO: UPX is not available.
93 INFO: Processing hook hook-os
218 INFO: Processing hook hook-time
233 INFO: Processing hook hook-cPickle
296 INFO: Processing hook hook-_sre
437 INFO: Processing hook hook-cStringIO
530 INFO: Processing hook hook-encodings
546 INFO: Processing hook hook-codecs
967 INFO: Processing hook hook-httplib
983 INFO: Processing hook hook-email
1108 INFO: Processing hook hook-email.message
1345 WARNING: library python%s%s required via ctypes not found
1486 INFO: Extending PYTHONPATH with C:\Users\Hp\PycharmProjects\GISdev
1486 INFO: checking Analysis
1486 INFO: building Analysis because out00-Analysis.toc non existent
1486 INFO: running Analysis out00-Analysis.toc
1486 INFO: Adding Microsoft.VC90.CRT to dependent assemblies of final executable
1611 INFO: Searching for assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_none ...
1611 INFO: Found manifest C:\Windows\WinSxS\Manifests\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91.manifest
1611 INFO: Searching for file msvcr90.dll
1611 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91\msvcr90.dll
1611 INFO: Searching for file msvcp90.dll
1611 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91\msvcp90.dll
1611 INFO: Searching for file msvcm90.dll
1611 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91\msvcm90.dll
1704 INFO: Analyzing C:\Python27\lib\site-packages\PyInstaller\loader\_pyi_bootstrap.py
1720 INFO: Processing hook hook-os
1736 INFO: Processing hook hook-site
1750 INFO: Processing hook hook-encodings
1861 INFO: Processing hook hook-time
1875 INFO: Processing hook hook-cPickle
1953 INFO: Processing hook hook-_sre
2095 INFO: Processing hook hook-cStringIO
2203 INFO: Processing hook hook-codecs
2625 INFO: Processing hook hook-httplib
2625 INFO: Processing hook hook-email
2750 INFO: Processing hook hook-email.message
3000 WARNING: library python%s%s required via ctypes not found
3233 INFO: Processing hook hook-pydoc
3296 INFO: Analyzing C:\Python27\lib\site-packages\PyInstaller\loader\pyi_importers.py
3405 INFO: Analyzing C:\Python27\lib\site-packages\PyInstaller\loader\pyi_archive.py
3514 INFO: Analyzing C:\Python27\lib\site-packages\PyInstaller\loader\pyi_carchive.py
3623 INFO: Analyzing C:\Python27\lib\site-packages\PyInstaller\loader\pyi_os_path.py
3623 INFO: Analyzing GIS_Helper.py
3655 INFO: Processing hook hook-PIL
3687 INFO: Processing hook hook-PIL.Image
3873 INFO: Processing hook hook-scipy.sparse.csgraph
4123 INFO: Processing hook hook-matplotlib
5046 INFO: Processing hook hook-parser
5171 INFO: Processing hook hook-distutils
5599 INFO: Processing hook hook-setuptools
5894 INFO: Processing hook hook-sysconfig
5925 INFO: Processing hook hook-xml
5987 INFO: Processing hook hook-xml.sax
6020 INFO: Processing hook hook-pyexpat
6098 INFO: Processing hook hook-win32com
6098 INFO: Processing hook hook-win32com.client
6115 INFO: Processing hook hook-pythoncom
6177 INFO: Processing hook hook-pywintypes
6302 INFO: Processing hook hook-win32ui
6832 WARNING: library libc.so.1 required via ctypes not found
9233 INFO: Processing hook hook-pytz
16182 INFO: Processing hook hook-matplotlib.backends
16579 INFO: Processing hook hook-_tkinter
16737 INFO: checking Tree
16737 INFO: building because out00-Tree.toc missing or bad
16737 INFO: building Tree out00-Tree.toc
17000 INFO: checking Tree
17000 INFO: building because out01-Tree.toc missing or bad
17000 INFO: building Tree out01-Tree.toc
17875 INFO: Analyzing hidden import 'scipy.linalg.cython_blas'
17875 INFO: Analyzing hidden import 'scipy.linalg.cython_lapack'
17875 INFO: Analyzing hidden import 'scipy.special._ufuncs_cxx'
17875 INFO: Analyzing hidden import 'multiprocessing'
17875 INFO: Analyzing hidden import 'multiprocessing.process'
17875 INFO: Hidden import 'ctypes.util' has been found otherwise
17875 INFO: Hidden import 'codecs' has been found otherwise
17875 INFO: Hidden import 'encodings' has been found otherwise
17875 INFO: Looking for run-time hooks
17875 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_PIL_Image.py
17984 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_Tkinter.py
18093 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_mplconfig.py
18218 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_mpldata.py
18312 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_pkgres.py
18500 INFO: Analyzing rthook C:\Python27\lib\site-packages\PyInstaller\loader\rthooks\pyi_rth_win32comgenpy.py
21330 INFO: Adding Microsoft.VC90.MFC to dependent assemblies of final executable
21455 INFO: Searching for assembly x86_Microsoft.VC90.MFC_1fc8b3b9a1e18e3b_9.0.21022.8_none ...
21455 INFO: Found manifest C:\Windows\WinSxS\Manifests\x86_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.21022.8_none_b81d038aaf540e86.manifest
21455 INFO: Searching for file mfc90.dll
21455 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.21022.8_none_b81d038aaf540e86\mfc90.dll
21455 INFO: Searching for file mfc90u.dll
21455 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.21022.8_none_b81d038aaf540e86\mfc90u.dll
21455 INFO: Searching for file mfcm90.dll
21455 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.21022.8_none_b81d038aaf540e86\mfcm90.dll
21455 INFO: Searching for file mfcm90u.dll
21455 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.mfc_1fc8b3b9a1e18e3b_9.0.21022.8_none_b81d038aaf540e86\mfcm90u.dll
23396 INFO: Searching for assembly x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.30729.6161_none ...
23398 INFO: Found manifest C:\Windows\WinSxS\Manifests\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57.manifest
23398 INFO: Searching for file msvcr90.dll
23398 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57\msvcr90.dll
23398 INFO: Searching for file msvcp90.dll
23398 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57\msvcp90.dll
23398 INFO: Searching for file msvcm90.dll
23398 INFO: Found file C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57\msvcm90.dll
23736 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
24282 INFO: Using Python library C:\Windows\system32\python27.dll
24542 INFO: Warnings written to C:\Users\Hp\PycharmProjects\GISdev\build\GIS_Helper2\warnGIS_Helper2.txt
24572 INFO: checking PYZ
24573 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing
24573 INFO: building PYZ (ZlibArchive) out00-PYZ.toc
30790 INFO: checking PKG
30792 INFO: rebuilding out00-PKG.toc because out00-PKG.pkg is missing
30792 INFO: building PKG (CArchive) out00-PKG.pkg
56437 INFO: checking EXE
56438 INFO: building because out00-EXE.toc missing or bad
56438 INFO: building EXE from out00-EXE.toc
57085 INFO: Appending archive to EXE C:\Users\Hp\PycharmProjects\GISdev\dist\GIS_Helper.exe`

Answer 1

The enclosed traceback log suggests that the root cause of the error is in Python 's built-in multiprocessing module:

Traceback (most recent call last):
  ...
  ...
  ...
  File "C:\Users\Hp\PycharmProjects\GISdev\build\gis_helper\out00-PYZ.pyz\multiprocessing", line 65, in <module>
  ImportError: No module named util

Indeed, multiprocessing 's __init__.py file has in line 65 the following import statement : from multiprocessing.util import SUBDEBUG, SUBWARNING .

Try specifying multiprocessing.util as a hidden import and see whether it solves the issue.