Passing opaque data to C++ callback from Python

Question

I'm embedding Python (using boost::python) into an application plugin that uses callbacks. Essentially, I want to do something like:

In Python (say test.py):

def do_something():
  ...

register_callback(do_something)

And on the C++ side I register the register_callback() function:

void register_callback(boost::python::object& o)
{
    // Access some persistent state information
}

BOOST_PYTHON_MODULE(foo)
{
  boost::python::def("register_callback", register_callback);
}

void library_entry()
{
  PyImport_AppendInittab("foo", initfoo);

  PyInitialize();

  // Create some persistent state information

  boost::python::exec_file("test.py", ...);
}

The issue here is that I need to create the Python context and store things away in an opaque pointer I return to the application. And when Python calls back into the C++, I need to recover that opaque value. For example:

Application -- (calls) --> my C++ library -- (runs) --> Python script -- (calls) --> my C++ library

For that last step, I need to recover some opaque data. Further, I need that opaque data to be persistent. The calls into my C++ library are callbacks from the application. The issue I'm having is trying to figure out how to pass that state information to the C++ register_callback() function. I've tried something like:

namespace bp = boost::python;

class state_info_t
{
};

void register_callback(std::shared_ptr<state_info_t>& state, bp::object& o);
{
  // Access some persistent state information
}

// Create some persistent state information
std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>();

PyImport_AppendInittab("foo", initfoo);

Py_Initialize();

std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__"))));
bp::object main_namespace = main_module->attr("__dict__");
std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo")));

main_namespace["foo"] = *foo_module;

bp::scope foo_scope(*foo_module);

// Both of these fail with _a lot_ of errors, most related to "no matching function call to 'get_signature'
bp::def("register_callback",
        [&](bp::object& o) { register_callback(state, o); },
        bp::arg("func"));

bp::def("register_callback",
        std::bind(register_callback, state, std::placeholders::_1),
        bp::arg("func"));

The other thought I had was to store the persistent data in the module's dictionary. But I don't know how to recover it in the callback. For example:

// Create some persistent state information
std::shared_ptr<state_info_t> state = std::make_shared<state_info_t>();

PyImport_AppendInittab("foo", initfoo);

Py_Initialize();

std::shared_ptr<bp::object> main_module = std::make_shared<bp::object>(bp::handle<>(bp::borrowed(PyImport_AddModule("__main__"))));
bp::object main_namespace = main_module->attr("__dict__");
std::shared_ptr<bp::object> foo_module = std::make_shared<bp::object>(bp::handle<>(PyImport_ImportModule("foo")));
bp::object foo_namespace = main_module->attr("__dict__");

main_namespace["foo"] = *foo_module;
foo_namespace["state"] = bp::handle<>(state); // Whatever the appropriate wrapper is

Then in C++ side of register_callback:

void register_callback(bp::object& o)
{
  // How do I extract "state" from the context?
}

I'm not keen on this last one, since it exposes the state information to the script.

I don't want to make the state information global since there may be multiple running Python instances. And regardless, with multiple Python instances I still need a way to determine which Python instance is running to select the appropriate state information.

Answer 1

I think I found a way to accomplish what I want.

Rather than actually create a module, I can create a class, then put an instance of that class into sys.modules (see this answer ).

So I do something like this:

namespace bp = boost::python;

class foo
{
  public:
    void register_callback(bp::object&);
};

...

// Register foo with Python (in the current scope)
bp::object foo_class = bp::class_<foo>("foo")
  .def("register_callback", &foo::register_callback);

// Get sys.modules
bp::object obj(bp::handle<>(bp::borrowed(PyImport_GetModuleDict())));

// Add an instance of foo to sys.modules
obj["foo"] = foo_class();

Now, in python:

import foo

def func():
  ...

foo.register_callback(func)

This seems to make a class instance appear as a module, and work as module.