dynamic_cast failure while calling C++ API from Python

Question

I am attempting to call a C++ API from python. Following is the code in the pseudo form.

class Engine { // Singleton Class which does a heavy duty work
public:
    static Engine* getEngine();
    bool init();
private:
    static Engine* m_instance;
    Engine();
};

// Following the code to wrap the call to engine to call from python
// Its only a simplified form

//engine_module.c
#include <Engine.h>
PyObject* initengine() {
    Engine* e = Engine::getInstance();
    e->init();
   // return the Py_BuildValue ...   
}

PyObject* initengine_module() {
//... init the module
}

// Python code
import engine_module
status = engine_module.init() 

Problem: The Engine class is in libengine.so and when it inits it fails because internally dynamic_cast fails. The Engine in turns loads other libraries using dlopen(). I searched the net to add RTDL_GLOBAL and -E option while linking but still its not resolved. Am I supposed to add the -E option while compiling the python itself? What could be the reason that Engine class works perfectly well when used in C++ code and does not work when used in python?

Edit 1: To clarify on the question from Cat++: libengine.so has many other classes which internally in Engine::init() uses dynamic_cast<>. The classes involved in dynamic_cast are not exposed to python at all. Only the Engine::init() is exposed.

Edit 2: The platform is Red Hat Linux and the compiler is Intel

Answer 1

The question is how and when the dynamic library are loaded. The code you show in engine_module.c references Engine , so the library with Engine will be automatically loaded before any of the initialization code in engine_module.c is executed. Similarly, any libraries used by Engine will be loaded before the library with Engine is loaded. All of which will be loaded using the flags Python used to load its interface module. ( RTDL_LOCAL is my guess.) Any dlopen you invoke later will find that the module has already been loaded, and ignore the request—including ignoring any options to dlopen you might have passed.

The way we solved this was to create a special loader module, which contained no direct references to any of the other modules. Python loads this module, which implements the initxxx function. The initxxx function explicitly loads all of the other modules needed, in a dependency determined order. (If A used B, B will be loaded before A.) With RTDL_GLOBAL , of course. In your case, this would be the libraries used by Engine , then Engine . The last module to be loaded would be the one with the Python interface; we put the Python initialization code in the constructor of a static object, so it would be executed automatically when the object was loaded, but you can also put it in a named function, as long as you get the address of this function using dlsym , rather than declaring it extern in any fashion. The important thing is to ensure that all of the libraries are first loaded by your explicit dlopen , and not implicitly as a result of an undefined external in some library loaded earlier.

Answer 2

My best guess is that your are trying to apply dynamic_cast to some non-polymorphic classes and that's where it fails. Remember that dynamic_cast works only for classes with at least one virtual method (destructor counts).