openCV to dlib - Mat to array2d

Question

I'm successfully opening and displaying a .avi video using OpenCV and I need this to go through OpenCV because I want to learn how to make OpenCV and dlib communicate.

For my understanding, a Mat has to be converted into an array2d in order to be processed by dlib so here's my first attempt:

cv::VideoCapture cap("/home/francesco/Downloads/05-1.avi");
cv::namedWindow("UNLTD", CV_WINDOW_AUTOSIZE);

while(1)
{
    cv::Mat temp;
    cv_image<bgr_pixel> cimg(temp);

    std::vector<rectangle> faces = detector(cimg);
    cout << faces.size() << endl;

    cv::imshow("UNLTD", temp);
 }

This returns the error

Error detected in file /usr/local/include/dlib/opencv/cv_image.h.
Error detected in function dlib::cv_image<pixel_type>::cv_image(cv::Mat) [with pixel_type = dlib::bgr_pixel].

Failing expression was img.depth() == cv::DataType<typename pixel_traits<pixel_type>::basic_pixel_type>::depth && img.channels() == pixel_traits<pixel_type>::num.
The pixel type you gave doesn't match pixel used by the open cv Mat object.
     img.depth():    0
     img.cv::DataType<typename pixel_traits<pixel_type>::basic_pixel_type>::depth: 0
     img.channels(): 1
     img.pixel_traits<pixel_type>::num: 3

I tried swapping bgr_pixel to rgb_pixel but without any luck. Looking around the internet somebody mentioned that the img.depth() is zero, therefore I should use unsigned char instead of rgb_pixel.

First thing: my video is playing in colors, so it does have 3 channels, I don't understand why it should be interpreted as a 1 channel image.

The strange thing is that, making that change from rgb_pixel to unsigned char, makes the software work but ZERO faces are detected on that video stream (that is the video of a guy talking and the face on the same video is detected with no problems by dlib on python.

I don't understand what I'm doing wrong

Answer 1

In your code, the temp is empty because you have not fed any frame from the video capture to it. Conversion of cv::Mat to dlib::array2d is also not correct. Please see this post for more information.

You may try:

cv::VideoCapture cap("/home/francesco/Downloads/05-1.avi");
cv::namedWindow("UNLTD", CV_WINDOW_AUTOSIZE);
dlib::frontal_face_detector  detector = dlib::get_frontal_face_detector();

while(1)
{
    cv::Mat temp;
    cap >> temp;

    dlib::array2d<bgr_pixel> dlibFrame;
    dlib::assign_image(dlibFrame, dlib::cv_image<bgr_pixel>(temp));

    std::vector<rectangle> faces = detector(dlibFrame);
    cout << faces.size() << endl;

    cv::imshow("UNLTD", temp);
}