C++ application crashes when embedded Python interpreter tries to import external module a second time

Question

If I import an external module twice in different pybind11::scoped_interpreter sessions, the application crashes in eval.h in function eval at following line:

PyObject *result = PyRun_String(buffer.c_str(), start, global.ptr(), local.ptr());

with

Exception thrown at 0x00007FFD710C4E0C (multiarray.cp36-win_amd64.pyd) in pybind-test.exe: 0xC0000005: Access violation writing location 0x000000000000000A.

Reproducible example code

namespace py = pybind11;
void test() {
    try {
        py::scoped_interpreter guard{};
        py::object mainScope = py::module::import("__main__").attr("__dict__");
        py::exec(
            "import numpy\n",
            mainScope);
    }
    catch (py::error_already_set const &pythonErr) {  std::cout << pythonErr.what(); }
}
int main() {
    test();   // Runs fine
    test();   // Crashes at py::exec
}

I feel like this has to do with the comment in pybind11's embed.h:

The interpreter can be restarted by calling initialize_interpreter again. Modules created using pybind11 can be safely re-initialized. However, Python itself cannot completely unload binary extension modules and there are several caveats with regard to interpreter restarting. All the details can be found in the CPython documentation. In short, not all interpreter memory may be freed, either due to reference cycles or user-created global data.

So there is no way to call the Python interpreter twice? I have a python file containing helper numpy functions that I need to call at different points of algorithm execution from C++. Does this mean I can't do that?

Answer 1

Paraphrasing from the discussion at pybind11 github repo.

Instead of using py::scoped_interpreter use py::initialize_interpreter and py::finalize_interpreter . Call the interpreter in between as many times as you like.

Caveat:, “the Python interpreter is not fully thread-safe In order to support multi-threaded Python programs, there's a global lock, called the global interpreter lock or GIL “.

Example usage:

namespace py = pybind11;
void test() {
    try {
        py::object mainScope = py::module::import("__main__").attr("__dict__");
        py::exec(
            "import numpy\n",
            mainScope);
    }
    catch (py::error_already_set const &pythonErr) {  std::cout << pythonErr.what(); }
}
int main() {
   py::initialize_interpreter();
    test();  
    test();   
    py::finalize_interpreter();
}

Answer 2

It might currently be better to never call PyFinalize, either directly, or via scoped_interpreter, according to this article https://www.boost.org/doc/libs/1_47_0/libs/python/todo.html#pyfinalize-safety .

Errors can result when re-loading modules after calling finalize and then initialize again. I ran into that when following the current approved answer https://stackoverflow.com/a/51069948/5994043 .

https://www.boost.org/doc/libs/1_47_0/libs/python/todo.html#pyfinalize-safety

PyFinalize Safety: Currently Boost.Python has several global (or function-static) objects whose existence keeps reference counts from dropping to zero until the Boost.Python shared object is unloaded. This can cause a crash because when the reference counts do go to zero, there's no interpreter. In order to make it safe to call PyFinalize() we must register an atexit routine which destroys these objects and releases all Python reference counts so that Python can clean them up while there's still an interpreter. Dirk Gerrits has promised to do this job.