pybind11 - std::vector member of boost::optional struct is emptied if any members of the struct are accessed

Question

I have a hierarchical data structure of 3 structs in C++, which I renamed in my minimum working example (mwe) House, Room, and Objects. Room is a boost::optional type and has a std::vector<Object> member containing all Objects in this room. The Objects are just a container for some numbers.

I am aware that this is overly complex for such information, but it is necessary in the original code and cannot be changed easily. I tried to change it into a std::experimental::optional since we do not use c++17, but this broke some parts in our c++ code and I don't know whether it would actually solve the problem.

In C++, I have no issues at all and the boost::optional and all member variables work perfectly. But after binding everything I run into the weird problem that the std::vector objects is emptied as soon as I access any member variables of the Room. This can be either the length, width, area or the objects itself as shown in the Python example. If the objects are accessed for the first time, they are actually returned normally, but when trying the second access they are gone as if the had been moved. The same behavior applies if you perform x = myHouse.kitchen.objects.copy() . The list is in x and can even be accessed multiple times, but the information in kitchen is immediately lost. Weirdly, this also only applies to the objects and all other double members can be accessed indefinitely.

For the convenience of compiling everything for the mwe and debugging, I stuffed everything into a single cpp file, which is obviously not the case in the original code. For the mwe, the c++ code is compiled and included via cppimport, but also manual compilation does not change anything.

Here is the mwe:

mwe.cpp:

#include <boost/optional.hpp>
#include <iostream>
#include <map>
#include <string>
#include <vector>
#include <pybind11/complex.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

/* implementing structs */
struct Object {
  Object() = default;
  Object(double price, double height) : price(price), height(height) {
  };

  double price = 0.0;
  double height = 0.0;
};

struct Room {
  Room() = default;

  double length;
  double width;
  std::vector<Object> objects; // this is the buggy vector

  double area() const {
    return length * width;
  };
};

struct House {
  House() = default;

  boost::optional<Room> bedroom;
  boost::optional<Room> kitchen;
  boost::optional<Room> livingroom;

  std::map<std::string, std::vector<Object>> getObjects() {
    std::map<std::string, std::vector<Object>> out;
    if (bedroom) {
      out.insert(std::make_pair("bedroom", bedroom->objects));
    }
    if (kitchen) {
      out.insert(std::make_pair("kitchen", kitchen->objects));
    }
    if (livingroom) {
      out.insert(std::make_pair("livingroom", livingroom->objects));
    }
    return out;
  };
};

/* everything works fine in C++ -> get data this way to have complete object map */
House initSomethingInCpp() {
  auto myHouse = House();
  myHouse.bedroom = Room();
  myHouse.kitchen = Room();

  myHouse.bedroom->length = 10.0;
  myHouse.bedroom->width = 2.0;
  myHouse.kitchen->length = 5.0;
  myHouse.kitchen->width = 3.0;

  std::vector<Object> bedroomObjects;
  std::vector<Object> kitchenObjects;

  Object closet = Object(100.0, 2.5);
  Object bed = Object(200.0, 1.0);
  Object oven = Object(500.0, 1.5);
  Object table = Object(50.0, 1.5);

  bedroomObjects.push_back(closet);
  bedroomObjects.push_back(bed);
  kitchenObjects.push_back(oven);
  kitchenObjects.push_back(table);

  myHouse.bedroom->objects = bedroomObjects;
  myHouse.kitchen->objects = kitchenObjects;

  return myHouse;
};

namespace pybind11 {
/* taking care of boost type */
namespace detail {
/* boost::optional */
template<typename T>
struct type_caster<boost::optional<T>> : optional_caster<boost::optional<T>> {};
} // namespace detail
} // namespace pybind11


/* binding structs */
void init_house(pybind11::module& main) {
  pybind11::class_<House> house(main, "House");
  house.def(pybind11::init<>());

  house.def_readwrite("bedroom", &House::bedroom);
  house.def_readwrite("kitchen", &House::kitchen);
  house.def_readwrite("livingroom", &House::livingroom);
  house.def("get_objects", &House::getObjects);
};

void init_room(pybind11::module& main) {
  pybind11::class_<Room> room(main, "Room");
  room.def(pybind11::init<>());
  room.def_readwrite("length", &Room::length);
  room.def_readwrite("width", &Room::width);
  room.def_readwrite("objects", &Room::objects);
  room.def_property_readonly("area", &Room::area);
};

void init_objects(pybind11::module& main) {
  pybind11::class_<Object> object(main, "Object");
  object.def(pybind11::init<>());
  object.def(pybind11::init<double, double>());

  object.def_readonly("price", &Object::price);
  object.def_readonly("heigth", &Object::height);
};

/* define module and bind init_in_cpp function */
PYBIND11_MODULE(mwe, m) {
 init_house(m);
 init_room(m);
 init_objects(m);
 m.def("init_something_in_cpp", &initSomethingInCpp);
};

execute.py:

import cppimport
#cppimport.set_quiet(False)
#cppimport.force_rebuild()
mod = cppimport.imp('mwe')

# get data
myHouse = mod.init_something_in_cpp()

print("\n")
print("all data is here")
objs = myHouse.get_objects()
print(objs)
print(myHouse.kitchen.area) # by accessing kitchen members, the objects list is emptied
print("\n")

print("kitchen objects are now missing")
objs = myHouse.get_objects()
print(objs)
print("but area still works:")
print(myHouse.kitchen.area) # everything but objects still works
print("\n")

print("also works directly with same variable")
print("bedroom objects are accessed:")
print(myHouse.bedroom.objects)
print("bedroom objects are accessed again:")
print(myHouse.bedroom.objects)

The execution gives the following output:

all data is here
{'bedroom': [mwe.Object object at 0x7fbc9c2a43f0, mwe.Object object at 0x7fbc9c2a4670], 'kitchen': [mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]}
15.0

kitchen objects are now missing
{'bedroom': [mwe.Object object at 0x7fbc9c2a4e70, mwe.Object object at 0x7fbc9c2a4eb0], 'kitchen': []}
but area still works:
15.0

also works directly with same variable
bedroom objects are accessed:
[mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]
bedroom objects are accessed again:
[]

Answer 1

Turns out this was actually a bug in pybind11 release 2.5 https://github.com/pybind/pybind11/issues/1919

It is fixed in the current master branch and future releases.