Link C++ program output with Python script

Question

I have a C++ program that uses some very specific method to calculate pairwise distances for a data set (30,000 elements). The output file would be 20 GB, and look something like this:

point1, point2, distancex
pointi, pointj, distancexx
.....

I then input the file to Python and use Python (NumPy) for clustering. It takes forever using Python to read the output file. Is there a way to connect the C++ program directly with my Python code to save time on I/O on the intermediate file? Maybe using SWIG?

Answer 1

I assume you have been saving ascii. You could modify your C++ code to write binary instead, and read it with numpy.fromfile .

For a more direct connection, you would wrap your C++ code as a library (remove main() and drive it from Python) using swig. This allows you to share the memory of arrays between C++ and Python.

You can use either Python's buffer protocol on the C++ side together with numpy.frombuffer on the Python side. Or you can use the numpy headers to directly work on numpy arrays in C++. Here is a small swig example project using the second method. (Disclaimer: I wrote it.)