Access violation when trying to read out object created in Python passed to std::vector on C++ side and then returned to Python

Question

Working with VS 2019, Python 3.7 64bit on Windows 10 and pybind11 2.4.3 I have run into the following problem:

When I create an object with a pybind11 py::class_ on the Python side and pass it directly to a method on the C++ side storing it in a std::vector, an attempt to read out the object later from Python results in an Access violation - no RTTI data . If the Python code stores the created object first in a Python variable to then pass it to C++ the subsequent access from Python works as intended.

I don't have much experience in using pybind11 and C++ so I am probably making a simple mistake, would appreciate any help on how to set up the C++ and pybind11 usage so that the Python workaround to use a variable is not needed and I don't get any access violation.

Here are some code details, the C++ code is

#include <iostream>

#include <vector>

#include <pybind11/pybind11.h>

using namespace std;


class XdmItem;

class XdmValue {

public:

    virtual XdmItem* itemAt(int n);

    virtual int size();

    void addXdmItem(XdmItem* val);


protected:
    std::vector<XdmItem*> values;
};

void XdmValue::addXdmItem(XdmItem* val) {
    values.push_back(val);
}

XdmItem* XdmValue::itemAt(int n) {
    if (n >= 0 && (unsigned int)n < values.size()) {
        return values[n];
    }
    return NULL;
}

int XdmValue::size() {
    return values.size();
}

class XdmItem : public XdmValue {

public:

    int size();

};

int XdmItem::size() {
    return 1;
}


namespace py = pybind11;

PYBIND11_MODULE(UseClassHierarchyAsPythonModule, m) {


    py::class_<XdmValue>(m, "PyXdmValue")
        .def(py::init<>())
        .def("size", &XdmValue::size)
        .def("item_at", &XdmValue::itemAt)
        .def("add_item", &XdmValue::addXdmItem);

    py::class_<XdmItem, XdmValue>(m, "PyXdmItem")
        .def(py::init<>())
        .def("size", &XdmItem::size);



#ifdef VERSION_INFO
    m.attr("__version__") = VERSION_INFO;
#else
    m.attr("__version__") = "dev";
#endif
}

the Python code that works flawlessly is

import UseClassHierarchyAsPythonModule

value = UseClassHierarchyAsPythonModule.PyXdmValue()

print(value, type(value))

print(value.size())

item = UseClassHierarchyAsPythonModule.PyXdmItem()

value.add_item(item)

print(value.size())

item0 = value.item_at(0)

print(item, type(item))

while the following code causes the Access violation - no RTTI data! :

import UseClassHierarchyAsPythonModule

value = UseClassHierarchyAsPythonModule.PyXdmValue()

print(value, type(value))

print(value.size())

value.add_item(UseClassHierarchyAsPythonModule.PyXdmItem())

print(value.size())

item0 = value.item_at(0)

print(item, type(item))

It gives

  Message=Access violation - no RTTI data!
  Source=C:\SomePath\AccessViolation.py
  StackTrace:
  File "C:\SomePath\AccessViolation.py", line 13, in <module>
    item0 = value.item_at(0)

If I enable native code debugging the stack trace include the pybind C++ code and is

>   UseClassHierarchyAsPythonModule.pyd!pybind11::polymorphic_type_hook<XdmItem,void>::get(const XdmItem * src, const type_info * & type) Line 818  C++
    UseClassHierarchyAsPythonModule.pyd!pybind11::detail::type_caster_base<XdmItem>::src_and_type(const XdmItem * src) Line 851 C++
    UseClassHierarchyAsPythonModule.pyd!pybind11::detail::type_caster_base<XdmItem>::cast(const XdmItem * src, pybind11::return_value_policy policy, pybind11::handle parent) Line 871  C++
    UseClassHierarchyAsPythonModule.pyd!pybind11::cpp_function::initialize::__l2::<lambda>(pybind11::detail::function_call & call) Line 163 C++
    UseClassHierarchyAsPythonModule.pyd!pybind11::handle <lambda>(pybind11::detail::function_call &)::<lambda_invoker_cdecl>(pybind11::detail::function_call & call) Line 100   C++
    UseClassHierarchyAsPythonModule.pyd!pybind11::cpp_function::dispatcher(_object * self, _object * args_in, _object * kwargs_in) Line 624 C++
    [External Code] 
    AccessViolation.py!<module> Line 13 Python

Any idea what is wrong in my C++/pybind11 use?

Answer 1

PyBind11 can auto-cast using an RTTI trick (that polymorphic_type_hook ; it's the same as I do in cppyy: you make a fake base class, cast the given address to the fake base, then read the RTTI to get the actual name, then do a lookup of the Python-proxy and apply a base to derived offset as needed). If the Python code creates the object first, it is found later by address (this to guarantee object identity), so no cast happening.

For that auto-cast to work properly, you really need a virtual destructor to guarantee (per the C++ standard) proper placement of the RTTI in the dll. I don't see in your base class (XdmValue).

(Aside, specific to Windows, I also always export the RTTI root node from the main application to guarantee there is only one. But if so, that should be the Python interpreter doing, or the first module, so I don't think that applies here. Also, I'm assuming of course you enable RTTI when building.)

Answer 2

Note: I'm not by far a PyBind11 expert, I just read the question and tried to figure out what the cause might be.
My guess is that the difference is that in the case where it doesn't work, the Python object is created just before add_item call (so is the C++ wrapped one) then just after the call it's being garbage collected (and together with it the C++ wrapped one), yielding Undefined Behavior (an invalid pointer).
Conversely, in the case it works, the object is not being garbage collected as it's "saved" in item (its refcount is greater than 0), and therefore the C++ wrapped object is also present. A delete item just after value.add_item(item) should reproduce the faulty behavior.

According to [ReadTheDocs.PyBind11]: Functions - Keep alive :

In general, this policy is required when the C++ object is any kind of container and another object is being added to the container. keep_alive<Nurse, Patient> indicates that the argument with index Patient should be kept alive at least until the argument with index Nurse is freed by the garbage collector.

So, the solution is to make the UseClassHierarchyAsPythonModule.PyXdmItem object persistent until the container is destroyed (note that this might keep objects in memory longer than expected, there might be a cleaner way to achieve this), and that is by specifying in add_item :

...

.def("add_item", &XdmValue::addXdmItem, py::keep_alive<1, 2>());