Calling python from C++, through callback, with variable buffer

Question

I managed to write code such that I can call python from C++ through a callback. ctypes was used.

The callback setup in Python is something like this:

proto = ctypes.CFUNCTYPE( ctypes.c_char_p, ctypes.POINTER( ctypes.c_char))

def parser( query, result):
   # Parsing code

fp = proto( parser_func)

---

This works, but I am passing a fixed 64k buffer from C++. That won't be acceptable: I need to handle a variable buffer. The only solution we have come up with was to add another callback, in which we return a handle to a python object. We then pass that into the parser callback. And then maybe we need a third function such that Python can reclaim the memory, or something like that

Looking for a simplest method to do this if anyone has ideas

Answer 1

I ended up implementing two callbacks. One calculates the length ( by parsing the query, and returning the length.) The other parses exactly like the first, but returns the parse output via the buffer that the client can now supply, knowing the length. It's not efficient, but it worked unlike everything else I tried. Thanks for the suggestions