从 C++ 函数调用 python lambda 时处理 GIL

Question

The question

Is pybind11 somehow magically doing the work of PyGILState_Ensure() and PyGILState_Release() ? And if not, how should I do it?

More details

There are many questions regarding passing a python function to C++ as a callback using pybind11, but I haven't found one that explains the use of the GIL with pybind11.

The documentation is pretty clear about the GIL:

[...] However, when threads are created from C (for example by a third-party library with its own thread management), they don't hold the GIL, nor is there a thread state structure for them.

If you need to call Python code from these threads (often this will be part of a callback API provided by the aforementioned third-party library), you must first register these threads with the interpreter by creating a thread state data structure, then acquiring the GIL, and finally storing their thread state pointer, before you can start using the Python/C API.

I can easily bind a C++ function that takes a callback:

py::class_<SomeApi> some_api(m, "SomeApi"); 
some_api
    .def(py::init<>())
    .def("mode", &SomeApi::subscribe_mode, "Subscribe to 'mode' updates.");

With the corresponding C++ function being something like:

void subscribe_mode(const std::function<void(Mode mode)>& mode_callback);

But because pybind11 cannot know about the threading happening in my C++ implementation, I suppose it cannot handle the GIL for me. Therefore, if mode_callback is called by a thread created from C++, does that mean that I should write a wrapper to SomeApi::subscribe_mode that uses PyGILState_Ensure() and PyGILState_Release() for each call?

This answer seems to be doing something similar, but still slightly different: instead of "taking the GIL" when calling the callback, it seems like it "releases the GIL" when starting/stopping the thread. Still I'm wondering if there exists something like py::call_guard<py::gil_scoped_acquire>() that would do exactly what I (believe I) need, ie wrapping my callback with PyGILState_Ensure() and PyGILState_Release() .

Answer 1

In general

pybind11 tries to do the Right Thing and the GIL will be held when pybind11 knows that it is calling a python function, or in C++ code that is called from python via pybind11. The only time that you need to explicitly acquire the GIL when using pybind11 is when you are writing C++ code that accesses python and will be called from other C++ code, or if you have explicitly dropped the GIL.

std::function wrapper

The wrapper for std::function always acquires the GIL via gil_scoped_acquire when the function is called , so your python callback will always be called with the GIL held, regardless which thread it is called from.

If gil_scoped_acquire is called from a thread that does not currently have a GIL thread state associated with it, then it will create a new thread state . As a side effect, if nothing else in the thread acquires the thread state and increments the reference count, then once your function exits the GIL will be released by the destructor of gil_scoped_acquire and then it will delete the thread state associated with that thread .

If you're only calling the function once from another thread, this isn't a problem. If you're calling the callback often, it will create/delete the thread state a lot, which probably isn't great for performance. It would be better to cause the thread state to be created when your thread starts (or even easier, start the thread from Python and call your C++ code from python).

Answer 2

Firstly, the documentation for the current version of python is here (unless you are targeting python 2.x?)

How python handles the GIL in extensions

The general setting in python is as follows: In the python context, the GIL is held. Thus, when calling into an extension, the GIL is also held at the beginning of the call. This is independent of whether you use plain C, pybind11, cython, or any other tool to generate the extension.

None of those tools magically release the GIL. The reason is twofold: First, devs have come to rely on the guarantees of GIL, which makes it possible to include libraries which are not thread-save themselves into python as extensions without any additional synchronization. Second, any manipulation of python objects without the GIL held will potentially lead to crashes.

Writing multi-threaded code

In terms of you writing multi-threaded code, problems occur whenever you interact with python within several threads. One workflow with respect to your function "mode" that avoids these issues is to

1. Convert all python data to C / C++ data in the original callback
2. Fork multiple threads to crunch the data, joining them afterwards
3. Create a python return value in the original thread after having finished your computations

This should not cause any problems with your python context. It will however lock up the interpreter, since the GIL is held.

So, if you want to crunch even more data by creating several python threads (using a ThreadPoolExecutor for instance) and calling into "mode" from several threads, only one call to "mode" will be executing at any given time.

This is of course rather unnecessary, since your worker threads in step 2 don't need to manipulate any python objects and the GIL does not need to be held during this time. Therefore, the GIL can be released during operations which do not need to interact with python. Thus, many C backends would release the GIL so the python interpreter can do other stuff in the meantime ([here] for example 2 ). The call would look like this:

1. Convert all python data to C / C++ data in the original callback
2. Release the GIL
3. Fork multiple threads to crunch the data, joining them afterwards
4. Acquire the GIL
5. Create a python return value in the original thread after having finished your computations

So, the process is reversed compared to how locks are normally used. Rather than acquiring a resource, working on it, and finally releasing it, a lock is released and reacquired after some work has been done.

However, there are some situations where the data crunching threads need to interact with python during the crunching. Think for instance of a numerical optimization tool which needs additional function evaluations at different times. Consequently, threads may need to acquire the GIL, do some python-related business and then release the GIL again. This is shown in the corresponding example in the pybind11 documentation.

It is for this reason that there exist the gil_scoped_release class (release upon creation, acquire upon destruction), to be used at the top level, and the gil_scoped_acquire (working in reverse), to be used in threads which are executed while the GIL is released. Again, there is no thread-related magic going on (afaik). You have to use these classes in your code when you interact with python from different threads.