Python Executable (by Pyinstaller in Virtual Conda Environment) DLL load failed

Question

I've created a simulation program in python which is working fine, when running it. The problem occurs when running the compiled executable of this program which was created with pyinstaller in a virtual environment from Anaconda. (I've to use this virtual environment, because I want to reduce the size of the executable by using no mkl packages). I get the error:

Traceback (most recent call last):
File "Main.py", line 2, in <module>
File "c:\anaconda3\envs\testing08\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "Simulationskern_einstufig_gen_2.py", line 1, in <module>
File "c:\anaconda3\envs\testing08\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\scipy\integrate\__init__.py", line 89, in <module>
File "c:\anaconda3\envs\testing08\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\scipy\integrate\quadrature.py", line 10, in <module>
File "c:\anaconda3\envs\testing08\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\scipy\special\__init__.py", line 640, in <module>
File "c:\anaconda3\envs\testing08\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 714, in load_module
module = loader.load_module(fullname)
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
[11472] Failed to execute script Main

To research the modul which is missing, I've edited the […]\\lib\\site-packages\\PyInstaller\\loader\\pyimod03_importers.py file and added two print commands which gave me the current state of the fullname and filename variables. After the fullname variable is “scipy.special._ufuncs” and filename variable is “[…]\\AppData\\Local\\Temp_MEI64402\\scipy.special._ufuncs.pyd” the executable crashed. To find out if the scipy.special._ufuncs file is missing, I did a copy of the “[…]\\AppData\\Local\\Temp_MEI64402\\” (it only exists a little seconds before the executable crashed). Unlike I suspected, I found the file scipy.special._ufuncs.pyd in the directory. To test if it is maybe corrupted or something, I executed the lines of the pyimod03_importers.py script manually in the conda prompt an everything works without an error. I'm really confused now and have no Idea what to do.

I've tried several combinations of hiddenimports in my Main.spec file, but none of them seemed to solve my problem. When compiling the program in my normal python environment, no error occurred, but the executable size becomes to big (260Mb), so this is not really a way to fix the problem. Reducing the size with UPX wasn't really helpful at all (260Mb to 180 Mb) which is still to big.

The code and the commands to create the executable are added below. Everything is run on a windows machine.

Skript: Main.py

#%% Import
import Simulationskern_einstufig_gen_2
from os.path import isfile
from os.path import isdir
from datetime import date
from class_Error import KennfeldPfadError
from class_Error import SystemSetupPfadError
from class_Error import SpeicherPfadError

#%% Funktionen 

def generate_filename(filenumber):
    datestring = str(date.today())
    if filenumber <= 9:
        numberstring = "00" + str(filenumber)
    elif (filenumber >= 10) & (filenumber <= 99):
        numberstring = "0" + str(filenumber)
    else:
        numberstring = str(filenumber)
    filestring = "result_" + datestring + "_" + numberstring
    return filestring

def run_simulation(data_base):
    if not isdir('D:\\'):
        raise SpeicherPfadError('Kritischer Fehler','Der angegebene Speicherpfad wurde nicht gefunden!') 
    testmode = False
    if testmode == False:
    data_base, ierr = Simulationskern_einstufig_gen_2.simulation(data_base)
    else:
        print("ACHTUNG: Testmodus ist aktiviert. Es werden keine Ergebnisse geschrieben!!!")
    #some testing routines

    if testmode == False:
        geschrieben = False
        filenumber = 1
        while geschrieben == False:
            filepath = 'D:\\' + generate_filename(filenumber)
            if not isfile(filepath + ".txt"):
                try:
                    # Ergebnis File schreiben
                    f = open(filepath + ".txt", "w")
                    for i in range(len(data_base["Ergebnisse"]["time"])):
                        f.write("%12.11e %12.11e %12.11e %12.11e %12.11e\n" %\
                                (data_base["Ergebnisse"]["time"][i],\
                                 data_base["Ergebnisse"]["x_masse1"][i],\
                                 data_base["Ergebnisse"]["v_masse1"][i],\
                                 data_base["Ergebnisse"]["x_masse2"][i],\
                                 data_base["Ergebnisse"]["x_masse2"][i]))
                    f.close()
                    geschrieben = True
                    print("Ergebnis " + filepath + " geschrieben")
                    break
                except FileNotFoundError:
                    raise SpeicherPfadError('Kritischer Fehler','Der angegebene Speicherpfad wurde nicht gefunden!')
            else:
                filenumber += 1

    return data_base

#%% Daten einlesen
try:
    from numpy import matrix
    from numpy.linalg import inv

    # Parameter
    m1  =  1   #[kg]
    m2  =  1   #[kg]
    c   = 10   #[N/m]
    k   =  1   #[Ns/m]

    data_base = dict()

    M = matrix(((m1,0),(0,m2)))
    data_base["M_inv"] = inv(M);
    data_base["C_sys"] = matrix(((c,-c),(-c,c)))
    data_base["K_sys"] = matrix(((k,-k),(-k,k)))
    data_base["F"] = matrix((1,0))
    data_base["abtastrate"] = 48000.0
    data_base["t_end"] = 100.0

    #%% Rechnen

    run_simulation(data_base)

except KennfeldPfadError as err:
    print(err.expression + ": " + err.message)
    input("Press Enter to Exit")
except SystemSetupPfadError as err:
    print(err.expression + ": " + err.message)
    input("Press Enter to Exit")
except SpeicherPfadError as err:
    print(err.expression + ": " + err.message)
    input("Press Enter to Exit")

Skript: Simulationskern_einstufig_gen_2.py

from scipy.integrate import ode
from numpy import zeros
from numpy import hstack
from numpy import asmatrix
from time import time
from c_module import Rechenkern

def simulation(data_base):
    ierr = 0
    #Indizierung
    index_masse1 = 0
    index_masse2 = 1

    #Übergabewerte definieren
    dim = len(data_base["M_inv"])

    F_ex = asmatrix(zeros(dim)).transpose(1,0)
    x_ex = asmatrix(zeros(dim)).transpose(1,0)
    v_ex = asmatrix(zeros(dim)).transpose(1,0)
    multi_ex1 = asmatrix(zeros(dim)).transpose(1,0)
    multi_ex2 = asmatrix(zeros(dim)).transpose(1,0)
    multi_ex3 = asmatrix(zeros(dim)).transpose(1,0)
    add_ex1 = asmatrix(zeros(dim)).transpose(1,0)
    add_ex2 = asmatrix(zeros(dim)).transpose(1,0)

    Kern = Rechenkern()
    Kern.set_params(data_base["M_inv"], -1.0*data_base["K_sys"], -1.0*data_base["C_sys"], dim, F_ex, x_ex, v_ex, multi_ex1, multi_ex2, multi_ex3, add_ex1, add_ex2, index_masse1, index_masse2)

    #Rechenfunktion für die Zustandswerte des nächsten Zeitschritts
    def f(t, y, Kern):
        v, a = Kern.calc_F(t, y)
        a = a[0:dim,0]
        v = v[0:dim,0]
        result_vektor = hstack((v,a))
        return result_vektor

    #Startwerte und Zeit festlegen
    y0 = asmatrix(zeros((1,dim*2))).transpose(1,0)
    t0 = 0.0
    t1 = data_base["t_end"]
    dt = 1/data_base["abtastrate"]

    #Solver Auswählen
    r = ode(f).set_integrator('vode',method='adams', with_jacobian=True)
    r.set_initial_value(y0, t0).set_f_params(Kern)

    #Listen zum Speichern der Ergebnisse vorbereiten
    counter = 0
    data_counter = 0
    data_base["Ergebnisse"] = dict()
    ergebnislaenge = int(t1*data_base["abtastrate"])
    data_base["Ergebnisse"]["time"] = zeros(ergebnislaenge)
    data_base["Ergebnisse"]["x_masse1"] = zeros(ergebnislaenge)
    data_base["Ergebnisse"]["x_masse2"] = zeros(ergebnislaenge)
    data_base["Ergebnisse"]["v_masse1"] = zeros(ergebnislaenge)
    data_base["Ergebnisse"]["v_masse2"] = zeros(ergebnislaenge)

    #Rechnen    
    start_time = time() 
    counter_time_a = start_time
    while r.successful() and r.t < t1:
      r.integrate(r.t+dt)
      if counter > 10000:
          counter_time_b = time()
          print("Zeit:",r.t, "Performance:", 10000.0/(counter_time_b-counter_time_a), "Hz")
          counter_time_a = counter_time_b
          counter = 0
      else:
          counter = counter+1
      try:
          data_base["Ergebnisse"]["time"][data_counter] = r.t
          data_base["Ergebnisse"]["x_masse1"][data_counter] = r.y[index_masse1]
          data_base["Ergebnisse"]["v_masse1"][data_counter] = r.y[index_masse1+dim]
          data_base["Ergebnisse"]["x_masse2"][data_counter] = r.y[index_masse2]
          data_base["Ergebnisse"]["v_masse2"][data_counter] = r.y[index_masse2+dim]
          data_counter = data_counter + 1
      except IndexError:
          print("Wert zu viel!")
    end_time = time()
    print("Dauer der Berechnung: " + str(end_time-start_time) + " s")
    return data_base, ierr

Skritp: c_module.pyx

from libc.math cimport sin, M_PI
cimport cython

cdef class Rechenkern():

    cdef double[:,:] M_inv
    cdef double[:,:] K_sys
    cdef double[:,:] C_sys
    cdef double[:,:] F_ex
    cdef double[:,:] x
    cdef double[:,:] v
    cdef double[:,:] multi_ex1
    cdef double[:,:] multi_ex2
    cdef double[:,:] multi_ex3
    cdef double[:,:] add_ex1
    cdef double[:,:] add_ex2
    cdef double[:,:] y_val_lb

    cdef int dim
    cdef int index_masse1
    cdef int index_masse2

    def set_params(self, double[:,:] M_inv, double[:,:] K_sys, double[:,:] C_sys, int dim, double[:,:] F_ex, double[:,:] x_ex, double[:,:] v_ex, double[:,:] multi_ex1, double[:,:] multi_ex2, double[:,:] multi_ex3, double[:,:] add_ex1, double[:,:] add_ex2, int index_masse1, int index_masse2):
        self.M_inv = M_inv
        self.K_sys = K_sys
        self.C_sys = C_sys
        self.F_ex = F_ex
        self.x = x_ex
        self.v = v_ex
        self.multi_ex1 = multi_ex1
        self.multi_ex2 = multi_ex2
        self.multi_ex3 = multi_ex3
        self.add_ex1 = add_ex1
        self.add_ex2 = add_ex2

        self.dim = dim
        self.index_masse1 = index_masse1
        self.index_masse2 = index_masse2

    @cython.boundscheck(False)
    @cython.cdivision(True)
    @cython.initializedcheck(False)
    def calc_F(self, double t, double[:] y):
        cdef double[:,:] a, stiffness, damping, system, total
        cdef int i
        for i in range(self.dim):
            self.x[i][0] = y[i]
            self.v[i][0] = y[i+self.dim]
        #Berechnen der äußeren Kräfte
        self.F_ex[self.index_masse1,0] = 1.0*sin(2.0*M_PI*20.0*t)
        #Bilden der Ableitungen
        stiffness = multiply(self.C_sys, self.x, self.dim, self.multi_ex1) #C_sys schon negiert als Parameter übergeben
        damping = multiply(self.K_sys, self.v, self.dim, self.multi_ex2) #K_sys schon negiert als Parameter übergeben
        system = add(stiffness, damping, self.dim, self.add_ex1)
        total = add(system, self.F_ex, self.dim, self.add_ex2)
        a = multiply(self.M_inv, total, self.dim, self.multi_ex3)
        return self.v, a

@cython.boundscheck(False)
@cython.cdivision(True)
cdef double[:,:] multiply(double[:,:] A, double[:,:] B, int dim, double[:,:] tmp_out):
    cdef int i,j
    cdef double s
    cdef double[:,:] out = tmp_out
    for i in range(dim): #Zeile
        s = 0.0
        for j in range(dim): #Spalte
            s += A[i,j]*B[j,0]
        out[i,0] = s
    return out

@cython.boundscheck(False)
@cython.cdivision(True)
cdef double[:,:] add(double[:,:] A, double[:,:] B, int dim, double[:,:] add_ex):
    cdef int i
    for i in range(dim):
        add_ex[i,0] = A[i,0] + B[i,0]
    return add_ex

Sktipt: class_Error.py

class Error(Exception):
    """Base class for exceptions in this module."""
    pass

class KennfeldPfadError(Error): 
    def __init__(self, expression, message):
        self.expression = expression
        self.message = message

class SystemSetupPfadError(Error): 
    def __init__(self, expression, message):
        self.expression = expression
        self.message = message

class SpeicherPfadError(Error): 
    def __init__(self, expression, message):
        self.expression = expression
        self.message = message

Skript: setup.py

import numpy

try:
    from setuptools import setup
    from setuptools import Extension
except ImportError:
    from distutils.core import setup
    from distutils.extension import Extension
from Cython.Build import cythonize   

setup(
    ext_modules = cythonize("c_module.pyx", annotate=True),include_dirs=[numpy.get_include()]
)

Skript: Main.spec

block_cipher = None

mkl_dlls = [('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\lib_arpack-.E27K2SF75AFV2JAJA2UCDNNFYUB5PRKN.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\lib_blas_su.SU77LBR3GCSGWOMPF2EOOUNRTVGOH5QW.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\lib_test_fo.JF5HTWMUPBXWGAYEBVEJU3OZAHTSVKCT.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libansari.R6EA3HQP5KZ6TAXU4Y4ZVTRPT7UVA53Z.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libbanded5x.GRJEXVLV2RZBSTTPFWQOQINCAR6VPZCI.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libbispeu.5N2XSD7URZS4WTOSLTOG4DDMA4HGB46U.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libblkdta00.G3LQYBY5GAOR5JJGRXYUWYKKDP4OKP2G.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libchkder.6HLXPVTQJEGRZGLI5DFRMNW3SS76BHP6.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libcobyla2.JEGTSUUFJ7DFXWZN5PAYZTTLBDATC4WD.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libd_odr.RVVJB6VEKU75HMLZ5VWMHFXDESN2K6A7.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdcosqb.K4J3XBR4PEETMRHZICUWW4LXG5UONZ34.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdcosqb.QRGA36MB6CFHWLQN6ETWARR4M4E6P3C2.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdcsrch.I2AOPDCXAPDRFNPWY55H5UE7XZSU5CVN.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdet.CF6EYFUDEYPH43I3Z6PMABQVSHN33WMB.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdfft_sub.L7RK2BXXS6VZ7XREQ326OSAIHHPLPCMK.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdfitpack.KTCF3EOE66VRDKN45KBQA4VBAIS552IF.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdgamln.HCGQFIEXCOZOROCO5Y4HSBVSKT76OCHM.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdop853.6TJTQZW3I3Q3QIDQHEOBEZKJ3NYRXI4B.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libdqag.XYDTZMPTRTODHGZ2AIXJLHZ6K7SZTGO7.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\liblbfgsb.OPNUZO4Z277J7J477DF55KBT2666RE7H.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libmvndst.FJYIV5KRHEB6IA4GIU6HFBYCEDXO3WN4.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libnnls.5LTQOLAJY5PFO6MOEXWNMRWVFRWHYHKT.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libopenblas.BNVRK7633HSX7YVO2TADGR4A5KEKXJAW.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libslsqp_op.NNY57ZXZ43A4RH3YWFA7BKHP5PC2K3I5.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libspecfun.PT6DS3HUOGYNSXUO4OUKK6ATA7B5KP2K.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libSTOPTEST.B2ZVH7DO5DVZYDS75ZSCHXD5G43HCYD4.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libvode.EZHESKPEGK6QRJZO64W6EHM63B4RVTTZ.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libwrap_dum.26QQ3L55FU6KXMGPYDA53KYMCT2TU35H.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\libwrap_dum.JPPGMGH3MZCSHXFMQQGFOSCGZ7ZEQKKE.gfortran-win_amd64.dll', '.'),
            ('C:\Anaconda3\Lib\site-packages\scipy\extra-dll\msvcp140.dll', '.')]

a = Analysis(['Main.py'],
         binaries=mkl_dlls,
         datas=[('C:\\Anaconda3\\Lib\\site-packages\\scipy\\special\\_ufuncs_cxx.cp36-win_amd64.pyd','.')],
         #, 'tkinter', 'matplotlib', 'fixtk'
         #datas=[('C:\\Anaconda3\\envs\\Testing08\\Lib\\site-packages\\scipy\\special\\_ufuncs_cxx.cp36-win_amd64.pyd','.'),('C:\\Anaconda3\\Lib\\site-packages\\scipy\\special\\_ufuncs.cp36-win_amd64.pyd','.'),],
         hiddenimports=['scipy._lib.messagestream', 'numpy', 'scipy', 'scipy.signal', 'scipy.signal.bsplines', 'scipy.special', 'scipy.special._ufuncs_cxx','scipy.special._ufuncs',
                        'scipy.linalg.cython_blas',
                        'scipy.linalg.cython_lapack',
                        'scipy.integrate',
                        'scipy.integrate.quadrature',
                        'scipy.integrate.odepack',
                        'scipy.integrate._odepack',
                        'scipy.integrate.quadpack',
                        'scipy.integrate._quadpack',
                        'scipy.integrate._ode',
                        'scipy.integrate.vode',
                        'scipy.integrate._dop', 'scipy._lib', 'scipy._build_utils','scipy.__config__',
                        'scipy.integrate.lsoda', 'scipy.cluster', 'scipy.constants','scipy.fftpack','scipy.interpolate','scipy.io','scipy.linalg','scipy.misc','scipy.ndimage','scipy.odr','scipy.optimize','scipy.setup','scipy.sparse','scipy.spatial','scipy.special','scipy.stats','scipy.version'],
         hookspath=[],
         runtime_hooks=[],
         excludes=[],
         win_no_prefer_redirects=False,
         win_private_assemblies=False,
         cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
exe = EXE(pyz,
      a.scripts,
      a.binaries,
      a.zipfiles,
      a.datas,
      name='Main',
      debug=False,
      strip=False,
      upx=True,
      runtime_tmpdir=None,
      console=True )

coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='Main')

I build the Executable with the following commands:

conda create --no-default-packages -n Tesing08 python=3.6.3

activate Tesing08

pip install numpy scipy pyinstaller cython

python setup.py build_ext --inplace

pyinstaller Main.spec --onefile

Answer 1

You must install pyinstaller with conda install pyinstaller in that virtual environment. After that, try building executable file again