How to use pybind11 to return a DataFrame?

Question

I am writing a Python module using pybind11 and Modern C++ .

How do I return a DataFrame from C++ to Python?

Answer 1

It is possible by returning an Apache Arrow table, which can be converted to a Python DataFrame with one line of Python.

For an example of an existing Python library that uses this:

• See the Turbodbc docs .
• See the Turbodbc github repo and the source code with methods to pass tables from C++ to Python using pybind11 .

Other links

• How to convert PyArrow table to Arrow table when interfacing between PyArrow in python and Arrow in C++

Answer 2

Sometimes, it's useful to do a quick'n'dirty transfer of a DataFrame from PyBind11/C++ to Python for logging purposes. We don't want speed, we want ease of use.

Construct a string that represents a.csv file in C++, return that, then convert that into a DataFrame on the Python side:

from io import StringIO
logCsv = 'A,B\n2.3,4.5\n'  # This string could be generated in PyBind11/C++.
LOGDATA = StringIO(logCsv)
df = pd.read_csv(LOGDATA, sep=",")
df

Output:

     A    B
0  2.3  4.5

Once we have this data in a DataFrame, we can save it in any format including Excel and Parquet. Once the data is in Excel, it becomes easier to debug.

If the cells are tab separated, then the data can be pasted straight from the log into Excel, and it will correctly divide into multiple cells.

from io import StringIO
logCsv = 'A\tB\n'
logCsv += '2.3\t4.5\n'  # This string could be generated in PyBind11/C++.
LOGDATA = StringIO(logCsv)
df = pd.read_csv(LOGDATA, sep="\t")
print(df)
# Can now paste output straight into Excel.