Boost not statically linking in to boost::python shared object

Question

I've created a wrapper for my application using boost::python.

This has worked so far by: (number of static libraries/source code) -> python_mapping.so

In this way my shared object is comprised of many static libs, including boost itself (notably boost_thread). I would assume that this so would contain ALL my application information, as I've statically linked everything in.

This compiles just fine.

ldd python_mapping.so
        librt.so.1 => /lib64/librt.so.1 (0x00002b7cbad37000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b7cbaf40000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b7cbb240000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b7cbb4c4000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b7cbb6d2000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b7cbb8ed000)
        /lib64/ld-linux-x86-64.so.2 (0x000000327ee00000)

However, when I run my example python application I get this run time linking error: undefined symbol: _ZTIN5boost6detail16thread_data_baseE

It seems that those boost libraries that linked in fine to the static library aren't actually there?

I made some progress on this. Apparently my shared object doesn't include a lot of symbols the compiler didn't think were being used (because it never saw my c++ objects ever being created, since the are created by instantiating Python Objects).

Kind of like:

//This is a class only created in python
#include "CPlusPlusClass.h"

PythonClass
{
       public:
           PythonClass() { }
       private:
           CPlusPlusClass _cplusplus;
};

//PythonMappings for PythonClass

#python file
import python_mapping

pythonClass = python_mapping.PythonClass() #This fails saying it can't find the symbol for CPlusPlusClass

Compiler will optimize out the CPlusCPlus class out because it never sees it actually being used, which is totally obnoxious. It does seem to keep the PythonClass itself (probably because of the Python Boost Mapping Macro.

You can get around that a few ways:

1. Link all your libraries with: -Xlinker --whole-archive
2. Create dummy uses of the libraries in your shared object

I was wondering if anyone can think of another solution, because it is really annoying to go through all the possible libraries and add them in with --whole-archive .

Answer 1

OK, since no one has responded, I've found the easiest (and only solution I know of) is to include EVERY static library you have as:

-Xlinker --whole-archive

You can, of course, link your libraries dynamically, which will require you to set LD_LIBRARY_PATH when executing your python application (though this wasn't an option for me for many libraries).

So, in this sense, explicit dynamic linking might be considered a more elegant solution.

Beyond that, if your libraries use other libraries:

python_mapping.so -> static linked in utility1.a -> static linked in utility2.a

If you've forgotten to link in utility1.a running the python application will let you know it can't find the symbols, however, it will NOT complain about utility2.a, and will have weird behavior when it reaches that part of the library. So... be careful and make sure you've explicitly linked in EVERYTHING.

Answer 2

You can use the linker parameter -rpath to embed a library search path, including "current directory" in your executable (g++ -Wl,-rpath,.) so you can specify exactly from where to load the shared libraries. This helps if you are moving your application around to other machines whose .so files are unknown. You'd have to also move over the .so files you explicitly specified. It's a compromise between static linking, and full dynamic linking because you won't actually be "sharing" the .so file with other apps on the box (you brought your own copies over), one of the memory saving benefits of using shared libraries.

This post is mostly to publicize the rarely used, but very useful -rpath parameter.