MATLAB MEX file on Linux fails to find symbols in libpython.2.7.so

Question

I've been working on a project to embed the Python interpreter into MATLAB as a MEX-file, using libpython2.7.so, but encounter a problem whenever I try to import Python modules implemented as extension modules. For example, attempting to import itertools results in the following error:

>> py_import itertools
ImportError: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: undefined symbol: PyTuple_Type
??? Error using ==> pymex_fns
Python exception inside py_import.

Error in ==> py_import at 24
    py_obj = pymex_fns(py_function_t.IMPORT, name);

This behavior occurs whether or not I clear LD_LIBRARY_PATH before the call to Py_Initialize() , and a call to ldd (for itertools.so , in this example) from within MATLAB doesn't result in any (not found) messages. Below, I've pasted the results of running MATLAB with LD_DEBUG=libs set, first with LD_LIBRARY_PATH set by MATLAB's startup, and then after running setenv('LD_LIBRARY_PATH', '') before attempting to import.

How can I fix this problem, and allow for the dynamic loading of extension modules?

With MATLAB-default LD_LIBRARY_PATH :

>> py_import itertools           
      3018: find library=libpython2.7.so.1.0 [0]; searching
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3018:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libpython2.7.so.1.0
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libpython2.7.so.1.0
      3018:  search cache=/etc/ld.so.cache
      3018:   trying file=/lib64/libpython2.7.so.1.0
      3018: 
      3018: find library=libutil.so.1 [0]; searching
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3018:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3018:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3018:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libutil.so.1
      3018:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libutil.so.1
      3018:  search cache=/etc/ld.so.cache
      3018:   trying file=/lib64/libutil.so.1
      3018: 
      3018: 
      3018: calling init: /lib64/libutil.so.1
      3018: 
      3018: 
      3018: calling init: /lib64/libpython2.7.so.1.0
      3018: 
      3018: 
      3018: calling init: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64
      3018: 
      3018: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64: error: symbol lookup error: undefined symbol: mexLibrary (fatal)
      3018: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: error: symbol lookup error: undefined symbol: PyTuple_Type (fatal)
??? Error using ==> pymex_fns
Python exception inside py_import.

Error in ==> py_import at 24
    py_obj = pymex_fns(py_function_t.IMPORT, name);

With LD_LIBRARY_PATH cleared:

>> py_import itertools           
      3125: find library=libpython2.7.so.1.0 [0]; searching
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libpython2.7.so.1.0
      3125:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libpython2.7.so.1.0
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libpython2.7.so.1.0
      3125:  search cache=/etc/ld.so.cache
      3125:   trying file=/lib64/libpython2.7.so.1.0
      3125: 
      3125: find library=libutil.so.1 [0]; searching
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64:/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64      (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../bin/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/../../sys/os/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/sys/os/glnxa64        (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/os/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/bin/glnxa64       (RPATH from file /usr/local/MATLAB/R2011a/bin/glnxa64/MATLAB)
      3125:   trying file=/usr/local/MATLAB/R2011a/bin/glnxa64/libutil.so.1
      3125:  search path=/usr/local/MATLAB/R2011a/extern/lib/glnxa64:/usr/local/MATLAB/R2011a/runtime/glnxa64:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64     (LD_LIBRARY_PATH)
      3125:   trying file=/usr/local/MATLAB/R2011a/extern/lib/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/runtime/glnxa64/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/server/libutil.so.1
      3125:   trying file=/usr/local/MATLAB/R2011a/sys/java/jre/glnxa64/jre/lib/amd64/libutil.so.1
      3125:  search cache=/etc/ld.so.cache
      3125:   trying file=/lib64/libutil.so.1
      3125: 
      3125: 
      3125: calling init: /lib64/libutil.so.1
      3125: 
      3125: 
      3125: calling init: /lib64/libpython2.7.so.1.0
      3125: 
      3125: 
      3125: calling init: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64
      3125: 
      3125: /home/cgranade/academics/software-projects/pymex-embed/src/pymex_fns.mexa64: error: symbol lookup error: undefined symbol: mexLibrary (fatal)
      3125: /usr/lib64/python2.7/lib-dynload/itertoolsmodule.so: error: symbol lookup error: undefined symbol: PyTuple_Type (fatal)

Answer 1

Try running the regular python both from inside MATLAB and on the system shell:

Shell

$ LD_DEBUG_OUTPUT=./ld_debug_out LD_DEBUG=all python -c 'import numpy'

MATLAB

>> !LD_DEBUG_OUTPUT=./ld_debug_out LD_DEBUG=all python -c 'import numpy'

and inspect the output files (maybe all is a bit too much info!). Note I imported numpy instead since itertools does not have a corresponding shared library on my installation..

---

One idea is to try to manually load the shared libraries in your MEX function before calling Py_Initialize() :

dlopen('libpython2.7.so', RTLD_GLOBAL|RTLD_LAZY);

---

I am on a Windows machine, but last time I played with this I had some problems as well. I remember that once imported, some C extension modules could not be unloaded and reloaded in the same session (during the lifetime of the hosting process where Python is embedded). See these remarks and related PEP .