Using libVLC to read RTSP video stream + convert data to OpenCV Mat

Question

I'm attempting to read an RTSP stream from a Sony FCB-EV7520 camera through an IP interface using the libVLC library and convert the data to the format used by OpenCV, namely the Mat type. I've been trying to find a good example of this for a couple of days, but the only results I've found so far are this and this . I followed the code in the first example, adapting it to the RTSP use case, but I have yet to retrieve any data in the Mat. However, from the terminal output I seem to achieve a connection to the camera. Do you see any obvious flaws in the code? Are there any other (easier) ways of achieving my goal? Any other libraries that could be used? Any help would be appreciated! The code im running is:

#include <stdio.h>
#include <vlc/vlc.h>
#include <stdint.h>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>

using namespace std;
using namespace cv;

struct VideoDataStruct {
    int param;
};

int done = 0; 
libvlc_media_player_t *mp;
unsigned int videoBufferSize = 0;
uint8_t *videoBuffer = 0;

void cbVideoPrerender(void *p_video_data, uint8_t **pp_pixel_buffer, int size) {

    // Locking

    // Locking a mutex here once I get the code to work.    

    if (size > videoBufferSize || !videoBuffer){

        printf("Reallocate raw video buffer\n");
        free(videoBuffer);
        videoBuffer = (uint8_t *) malloc(size);
        videoBufferSize = size;

    }

    *pp_pixel_buffer = videoBuffer;
}  

void cbVideoPostrender(void *p_video_data, uint8_t *p_pixel_buffer, int width, int height, int pixel_pitch, int size, int64_t pts) {

    // Unlocking

    // Unlocking the mutex here once I get the code to work.

}

static void handleEvent(const libvlc_event_t* pEvt, void* pUserData) {
    libvlc_time_t time;
    switch(pEvt->type){

        case libvlc_MediaPlayerTimeChanged:

            time = libvlc_media_player_get_time(mp);
            printf("MediaPlayerTimeChanged %lld ms\n", (long long)time);
            break;

        case libvlc_MediaPlayerEndReached:

            printf ("MediaPlayerEndReached\n");
            done = 1;
            break;

        default:

            printf("%s\n", libvlc_event_type_name(pEvt->type));

    }
}

int main(int argc, char* argv[]) {

    // VLC pointers: 

    libvlc_instance_t *inst;
    libvlc_media_t *m;
    void *pUserData = 0;
    VideoDataStruct dataStruct;

    // VLC options:

    char smem_options[1000];

    // RV24:

    sprintf(smem_options
            , "#transcode{vcodec=h264}:smem{"
            "video-prerender-callback=%lld,"
            "video-postrender-callback=%lld,"
            "video-data=%lld,"
            "no-time-sync},"
            , (long long int)(intptr_t)(void*)&cbVideoPrerender
            , (long long int)(intptr_t)(void*)&cbVideoPostrender
            , (long long int)(intptr_t)(void*)&dataStruct
           );

    const char * const vlc_args[] = {
        "-I", "dummy",            // Don't use any interface
        "--ignore-config",        // Don't use VLC's config
        "--extraintf=logger",     // Log anything
        "--verbose=2",            // Be verbose
        "--sout", smem_options    // Stream to memory
    };

    // We launch VLC

    inst = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);

    /* Create a new item */

    m = libvlc_media_new_location(inst, "rtsp://*****:*******@IP/videoinput_1/h264_1/media.stm");

    /* Create a media player playing environement */

    mp = libvlc_media_player_new_from_media (m);

    libvlc_event_manager_t* eventManager = libvlc_media_player_event_manager(mp);
    libvlc_event_attach(eventManager, libvlc_MediaPlayerTimeChanged, handleEvent, pUserData);
    libvlc_event_attach(eventManager, libvlc_MediaPlayerEndReached, handleEvent, pUserData);
    libvlc_event_attach(eventManager, libvlc_MediaPlayerPositionChanged, handleEvent, pUserData);

    libvlc_video_set_format(mp, "h264", 1920, 1080, 1920* 3 );

    /* play the media_player */
    libvlc_media_player_play (mp);

    while(1){

        if(videoBuffer){            // Check for invalid input

            Mat img = Mat(Size(1920, 1080), CV_8UC3, videoBuffer);  // CV_8UC3 = 8 bits, 3 chanels
            namedWindow("Display window", WINDOW_AUTOSIZE);     // Create a window for display.
            imshow("Display window", img);              // Show our image inside it.

        }
    }

    libvlc_release (inst);

}

I am running the code on Ubuntu 16.04 using OpenCV 3.2 and libVLC 2.2.5.1. If you need any additional information, just ask.

PS: I know that the stream is working as I can open it through the streaming interface of the VLC player. I also know that libVLC can decode the stream as I have successfully opened recorded mp4s of the stream.

Answer 1

Not a complete answer, but too long for a comment:

In cbVideoPrerender , videoBuffer is always allocated as needed by vlc.
However, in main , you have the loop while(1){ if (videoBuffer) { ... } } where videoBuffer is always true from the first time cbVideoPrerender has been called. This means that from then on, the loop is infinite and non-blocking, so there is no synchronization between the input of the video and processing and if you're just trying to get the first image, you will be too early.

Your first link suggests using cbVideoPostrender as a synchronisation point so you know the frame can be read and thus converted to the needed format. There it is done in the function itself, but you could have some mechanism with a condition variable (queue, event,...) to process the frame in another thread and pass the image to openCV.

By the way: using a variable by setting in one thread and reading it in another one without thread mechanisms (mutex, atomic) is usably a bad idea.