C++ 20 coroutines with PyBind11

Question

I'm trying to get a simple C++ 20 based generator pattern work with PyBind11. This is the code:

#include <pybind11/pybind11.h>
#include <coroutine>
#include <iostream>

struct Generator2 {
    Generator2(){}
    struct Promise;
    using promise_type=Promise;
    std::coroutine_handle<Promise> coro;
    Generator2(std::coroutine_handle<Promise> h): coro(h) {}
    ~Generator2() {
        if(coro)
            coro.destroy();
    }
    int value() {
        return coro.promise().val;
    }
    bool next() {
        std::cout<<"calling coro.resume()";
        coro.resume();
        std::cout<<"coro.resume() called";
        return !coro.done();
    }
    struct Promise {
        void unhandled_exception() {std::rethrow_exception(std::move(std::current_exception()));}
        int val;
        Generator2 get_return_object() {
            return Generator2{std::coroutine_handle<Promise>::from_promise(*this)};
        }
        std::suspend_always initial_suspend() {
            return {};
        }
        std::suspend_always yield_value(int x) {
            val=x;
            return {};
        }
        std::suspend_never return_void() {
            return {};
        }
        std::suspend_always final_suspend() noexcept {
            return {};
        }
    };
};


Generator2 myCoroutineFunction() {
    for(int i = 0; i < 100; ++i) {
        co_yield i;
    }
}

class Gen{
private:
    Generator2 myCoroutineResult;
public:
    
    Gen(){
        myCoroutineResult = myCoroutineFunction();
    }

    int next(){
        return (myCoroutineResult.next());
    }
};


PYBIND11_MODULE(cmake_example, m) {
    pybind11::class_<Gen>(m, "Gen")
            .def(pybind11::init())
            .def("next", &Gen::next);
}

However I'm getting an error:

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Could c++ coroutines, coroutine_handles, co_yield etc. be a low-level thing that is not supported by PyBind11 yet?

Answer 1

Even though PyBind11 does not support coroutines directly, your problem does not mix coroutine and pybind code since you are hiding the coroutine behind Gen anyway.

The problem is that your Generator2 type uses the compiler provided copy and move constructors.

This line:

myCoroutineResult = myCoroutineFunction();

Creates a coroutine handle when you call myCoroutineFunction , and puts it in the temporary Generator2 in the right hand side. Then, you initialize myCoroutineResult from the right hand side generator. All is well, but then the temporary gets destroyed. Your destructor checks whether the handle is valid or not:

~Generator2() {
    if(coro)
        coro.destroy();
}

But in your implementation, the coro member of the member generator gets copied from the temporary without resetting the temporary's coro member. So the coroutine itself gets destroyed once you initialize myCoroutineResult , and you are holding onto a dangling coroutine handle. Remember that std::coroutine_handle s behave like a raw pointer.

Essentially, you have a violation of the rule of 5. You have a custom destructor, but no copy/move constructors or assignment operators. Since you cannot copy construct a coroutine, you can ignore the copy constructors but you need to provide move constructors/assigment operators:

Generator2(Generator2&& rhs) : coro{std::exchange(rhs.coro, nullptr)} {
    // rhs will not delete our coroutine, 
    // since we put nullptr to its coro member
}

Generator2& operator=(Generator2&& rhs) {
    if (&rhs == this) {
        return *this;
    }
    if (coro) {
        coro.destroy();
    }
    coro = std::exchange(rhs.coro, nullptr);
    return *this;
}

Also, use member initialization list to initialize members instead of assigning them within the constructor body. So instead of this:

Gen(){
    myCoroutineResult = myCoroutineFunction();
}

Use this:

Gen() : myCoroutineResult{myCoroutineFunction()} {}

The reasoning can be seen even in this answer. The first one calls the assignment operator, which performs a bunch of additional work, whereas the second one calls the move constructor, which is as lean as it gets.