Python多处理不支持LD_LIBRARY_PATH进行动态加载

Question

I am working on a python 3.5 project that uses multiprocessing, the worker process needs to be able to call compiled MATLAB code. In order to not have to set the LD_LIBRARY_PATH environment variable before running python (it causes conflicts with libexpat), I want to have only this worker process using the altered LD_LIBRARY_PATH. I figured this would work fine since processes created with fork() are started with any environment changes made in the parent. I am using

matlab_runtime_paths = [
    '/usr/local/MATLAB/MATLAB_Runtime/v91/runtime/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/bin/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/sys/os/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/sys/opengl/lib/glnxa64'
]
system_lib_dir = '{}/lib'.format(sys.prefix)

lib_paths = matlab_runtime_paths + [system_lib_dir]
lib_paths_format = ':'.join(lib_paths)
os.putenv('LD_LIBRARY_PATH', lib_paths_format)

to set the environment variable in parent process, then later in the worker process I have

def matlab_worker(matlab_pipe_end):
    import service
    service.initialize_stub()

which crashes because it is unable to find the library properly. I can see that the environment variable is being set correct, if I add

def matlab_worker(matlab_pipe_end):
    os.system('echo $LD_LIBRARY_PATH')
    import service
    service.initialize_stub()

the variable is set, I can even do

def matlab_worker(matlab_pipe_end):
    os.system('ldd <path>/service.so')
    import service
    service.initialize_stub()

and see that all shared objects are resolved, yet python fails to honor this setting, crashing on the import. I'm assuming because the dynamic loader doesnt re-read the LD_LIBRARY_PATH variable after a fork (can that be correct? It doesnt sound correct but I cant find any documentation about it)

Is there any way to do what I want or is there something wrong with my code? I should note that this code works fine if the parent process is launched with LD_LIBRARY_PATH=... and that I have made sure there are no conflicting libraries that could be screwing up the process.

Answer 1

I'm assuming because the dynamic loader doesnt re-read the LD_LIBRARY_PATH variable after a fork

That's true, by the time you call putenv dynamic linker has already parsed LD_LIBRARY_PATH and built it's internal tables for resolving symbols. You won't be able to rebuild those without exec .

Alternatively, you can

• at startup, see if paths have been set and if they haven't, set them and re- exec the script
• change worker scripts to be standalone so that they can be exec -ed rather than fork -ed
• start your main app with LD_LIBRARY_PATH set
• manually load needed libraries via dlopen , using absolute paths