How to get the OpenCV image from Python and use it in C++ in pybind11?

Question

I'm trying to figure out how it is possible to receive an OpenCV image from a Python in C++. I'm trying to send a callback function, from C++ to my Python module, and then when I call a specific python method in my C++ app, I can access the needed image.

Before I add more details, I need to add that there are already several questions in this regard including :

1. how-to-convert-opencv-image-data-from-python-to-c
2. pass-image-data-from-python-to-cvmat-in-c
3. writing-python-bindings-for-c-code-that-use-opencv
4. c-conversion-from-numpy-array-to-mat-opencv

but none of them have anything about Pybind11 . In fact they are all using the PyObject (from Python.h header) with and without Boost.Python . So my first attempt is to know how it is possible in Pybind11 knowing that it has support for Numpy arrays, so it can hopefully make things much easier.

Also On the C++ side, OpenCV has two versions, 3.x and 4.x which 4.x as I've recently found, is C++11 compliant. on Python side, I used OpenCV 3.x and I'm on a crossroad of which one to choose and what implications it has when it comes to Pybind11 .

What I have tried so far: I made a quick dummy callback and tried passing a simple cv::Mat& like this :

#include <pybind11/embed.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>
namespace py = pybind11;
...

void cpp_callback1(bool i, std::string id, cv::Mat img)
{ 
    auto timenow = chrono::system_clock::to_time_t(chrono::system_clock::now());
    cout  <<"arg1: " << i << " arg2: " << id<<" arg3: " << typeid(img).name() <<" " << ctime(&timenow)<<endl;
}

and used it like this :

py::list callback_lst;
callback_lst.attr("append")(py::cpp_function(cpp_callback1));

py::dict core_kwargs = py::dict("callback_list"_a = callback_lst,
                                "debug_show_feed"_a = true);

py::object core_obj = core_cls(**core_kwargs);
core_obj.attr("start")();

but it fails with an exception on python part which says :

29/03/2020 21:56:47 : exception occured ("(): incompatible function arguments. The following argument types are supported:\n    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None\n\nInvoked with: True, '5', array([[[195, 217, 237],\n        [195, 217, 237],\n        [196, 218, 238],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       ...,\n\n       [[120, 129, 140],\n        [110, 120, 130],\n        [113, 122, 133],\n        ...,\n        [196, 209, 245],\n        [195, 207, 244],\n        [195, 207, 244]],\n\n       [[120, 133, 142],\n        [109, 121, 130],\n        [114, 120, 131],\n        ...,\n        [195, 208, 242],\n        [195, 208, 242],\n        [195, 208, 242]],\n\n       [[121, 134, 143],\n        [106, 119, 128],\n        [109, 114, 126],\n        ...,\n        [194, 207, 241],\n        [195, 208, 242],\n        [195, 208, 242]]], dtype=uint8)",) 
Traceback (most recent call last):
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 257, in start
    self._main_loop()
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 301, in _main_loop
    self._execute_callbacks(is_valid, name, frame)
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 142, in _execute_callbacks
    callback(*args)
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None

Invoked with: True, '5', array([[[195, 217, 237],
        [195, 217, 237],
        [196, 218, 238],
        ...,
        [211, 241, 255],
        [211, 241, 255],
        [211, 241, 255]],

       [[195, 217, 237],
        [195, 217, 237],
        [195, 217, 237],
        ...,

Using py::object or py::array_t<uint8_t> instead of cv::Mat doesn't cause any errors, but I can't seem to find a way to cast them back to a cv::Mat properly!

I tried to cast the numpy array into a cv::Mat as instructed in the comments but the output is garbage:

void cpp_callback1(bool i, std::string id, py::array_t<uint8_t>& img)
{ 
    auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    //py::buffer_info buf = img.request();
    cv::Mat img2(rows, cols, type, img.ptr());
    cv::imshow("test", img2);
}

results in :

It seems to me, the strides, or something in that direction is messed up that image is showing like this. what am I doing wrong here? I couldn't use the img.strides() though! when printed it using py::print, it shows 960 or something like that. So I'm completely clueless how to interpret that!

Answer 1

I ultimately could successfully get this to work thanks to @ DanMasek and this link :

void cpp_callback1(bool i, std::string id, py::array_t<uint8_t>& img)
{ 
    py::buffer_info buf = img.request();
    cv::Mat mat(buf.shape[0], buf.shape[1], CV_8UC3, (unsigned char*)buf.ptr);

    cv::imshow("test", mat);
}

note that the cast is necessary, or otherwise, you'd get a blackish screen only!
However, if somehow there was a way like py::return_value_policy that we could use to change the type of reference, so even though the python part ends, the c++ side wouldn't crash would be great.

side note :
it seems the ptr property exposed in the numpy array, is actually not a py::handle but a PyObject*& . I couldn't have a successful conversion and thus resorted to the solution I posted above. I'll update this answer, when I figure this out.

Update:

I found out, the arrays data holds a pointer to the underlying buffer and can be used easily as well. From <pybind11/numpy.h> L681:

/// Pointer to the contained data. If index is not provided, points to the
/// beginning of the buffer. May throw if the index would lead to out of bounds access.

So my original code that used img.ptr() , can work using img.data() like this :

void cpp_callback1(bool i, std::string id, py::array_t<uint8_t>& img)
{ 
    //auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    cv::Mat img2(rows, cols, type, (unsigned char*)img.data());
    cv::imshow("test", img2);
}

Answer 2

To convert between cv::Mat and np.ndarray , you can use pybind11_opencv_numpy .

Copy ndarray_converter.h and ndarray_converter.cpp to your project directory.

---

CMakeLists.txt

add_subdirectory(pybind11)
execute_process(COMMAND ${PYTHON_EXECUTABLE} -c "import numpy; print(numpy.get_include())" OUTPUT_VARIABLE NUMPY_INCLUDE OUTPUT_STRIP_TRAILING_WHITESPACE)
message(STATUS "NUMPY_INCLUDE: " ${NUMPY_INCLUDE})
include_directories(${NUMPY_INCLUDE})
pybind11_add_module(mymodule "cpp2py.cpp" "ndarray_converter.cpp")
target_link_libraries(mymodule PRIVATE ${OpenCV_LIBS})
target_compile_definitions(mymodule PRIVATE)

---

cpp2py.cpp

#include "ndarray_converter.h"

PYBIND11_MODULE(mymodule, m)
{
    NDArrayConverter::init_numpy();
    ...
}

Answer 3

This would be a generic conversion of an image with any number of channels and stride possibly different from the standard one (for example if the Mat has been obtained as a region of interest in a bigger matrix)

#include <pybind11/pybind11.h>

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    cv::Mat mat(img.shape(0), img.shape(1), CV_MAKETYPE(CV_8U, img.shape(2)),
                const_cast<uint8_t*>(img.data()), img.strides(0));

    cv::imshow("test", mat);
}

• img.shape(0) -> rows
• img.shape(1) -> cols
• img.shape(2) -> n_channels
• img.strides(0) -> stride in bytes between two neighboring pixelx in the same column