Using Boost.Python to build a shared lib and import it in Blender through Python

Question

What I currently try to achieve is building a python mapping of my C++ classes through Boost.Python. After this I want to use the resulting shared library in a blender add-on to be able to take advantage of already existing functionality coming from the mapped C++ classes.

I can already build my shared library and write sample scripts in python, which are using my library as well.

Everything fine here but the problem is that as soon as I try to use it in an add-on, Blender 2.74 keeps crashing all the time as soon as I add an import statement with this little hint in the crash report:

6   libboost_python.dylib           0x000000010aa7cc3e boost::python::detail::init_module(PyModuleDef&, void (*)()) + 30 (module.cpp:44)

In module.cpp inside of boost line 41-46:

BOOST_PYTHON_DECL PyObject* init_module(PyModuleDef& moduledef,       void(*init_function)())
{
    return init_module_in_scope(
        PyModule_Create(&moduledef),
        init_function);
}

My boost 1_58 is compiled using Python 3.4.2:

otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

Running the python3 bin from that directory gives me:

python3
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  5 2014, 20:42:22)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

The Blender Python console gives me:

PYTHON INTERACTIVE CONSOLE 3.4.2 (default, Nov 25 2014, 12:01:44)  [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]

Command History:     Up/Down Arrow
Cursor:              Left/Right Home/End
Remove:              Backspace/Delete
Execute:             Enter
Autocomplete:        Ctrl-Space
Zoom:                Ctrl +/-, Ctrl-Wheel
Builtin Modules:     bpy, bpy.data, bpy.ops, bpy.props, bpy.types, bpy.context, bpy.utils, bgl, blf, mathutils
Convenience Imports: from mathutils import *; from math import *
Convenience Variables: C = bpy.context, D = bpy.data
>>> 

My own project uses this python version as well. Also Blender uses this Python version.

I really don't know what to try next here since in standalone mode EVERYTHING works as expected. Also the fact that the crash even occurs as soon as I run a new script inside Blenders Text Editor having the import statement.

Anybody having experience with Boost.Python and Blender?

Thanks in advance

UPDATE: After building and running blender in debug mode, I got the following state:

This doesn't look right and describes the crash in the first place. Still searching for the cause...

Answer 1

Blender itself (as well as oiio and osl libs) are built using boost, ensure that the version you use to build boost-python matches the one used by blender. As blender has already loaded boost libs into ram, when you import boost-python it will use the already loaded boost libs not the ones you built boost-python with, if each is a different version it can cause problems, if each is built using different options it can also cause issues.

If you are building blender yourself also build OpenImageIO and OpenShadingLanguage so they use the same boost version. By building them yourself you can ensure they all load the version that you want to use. Check your cmake or sconscript settings point to the same boost lib files.

% ldd blender | grep boost
libboost_filesystem.so.1.55.0 => /usr/local/lib/libboost_filesystem.so.1.55.0 (0x806009000)
libboost_regex.so.1.55.0 => /usr/local/lib/libboost_regex.so.1.55.0 (0x80621e000)
libboost_system.so.1.55.0 => /usr/local/lib/libboost_system.so.1.55.0 (0x806525000)
libboost_thread.so.1.55.0 => /usr/local/lib/libboost_thread.so.1.55.0 (0x806728000)
libboost_date_time.so.1.55.0 => /usr/local/lib/libboost_date_time.so.1.55.0 (0x806942000)
libboost_wave.so.1.55.0 => /usr/local/lib/libboost_wave.so.1.55.0 (0x806b4f000)
libboost_locale.so.1.55.0 => /usr/local/lib/libboost_locale.so.1.55.0 (0x806e94000)
libboost_chrono.so.1.55.0 => /usr/local/lib/libboost_chrono.so.1.55.0 (0x814ee3000)

Answer 2

After a lot of trial and error I got it working! I'll try to give a step by step tutorial on how to do that for OSX. In the next couple of weeks I'll also try to get it working on windows and will update this answer as I succeed!

Step 1 Install python 3.4.2 from https://www.python.org/downloads/release/python-342/

Step 2 Checkout the blender git repo and the svn repo from blender (found at http://wiki.blender.org/index.php/Dev:Doc/Building_Blender/Mac ):

mkdir ~/blender-build
cd ~/blender-build
git clone http://git.blender.org/blender.git
cd blender
git submodule update --init --recursive
git submodule foreach git checkout master
git submodule foreach git pull --rebase origin master

Download external libs:

cd ~/blender-build
mkdir lib
cd lib
svn checkout https://svn.blender.org/svnroot/bf-blender/trunk/lib/darwin-9.x.universal

Step 3 Use CMake to create the XCode project for building blender Make sure you check the following settings for python: This will automatically configure to build blender using the installed python framework on your mac.

Step 4 Download and build boost with the Python 3.4.2 Here you first have to check which version blender uses. In the external libraries svn repository boost is contained as a pre build library at:

blender-build/lib/darwin-9.x.universal/README

For me this is boost 1.51: http://www.boost.org/users/history/version_1_51_0.html

Step 5 Build your shared library with Boost 1.51 and Python 3.14

Step 6 Build blender and make sure it is using the installed Python 3.4.2

Step 7 Copy your shared library into the addons folder of the blender.app Here a tiny small little hint: In order to let python recognize your shared library it has to have the extension .so rather than .dylib (for windows this is .pyd rather than .dll)!

Step 8 Recheck Check that your boost is compiled agains Python 3.4.2:

otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python3.dylib:
libboost_python3.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0) <- IMPORTANT
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

You can also check if your library can be used in a simple python script like test.py by executing the following script in a folder were your .so is contained:

import mylib
print('hello world!')

Run:

python3 test.py

Basically now you should also be able to execute this simple script inside of blender, which is the case for me.