OpenCV - Streaming H264 over RTSP using FFMPEG in version 3.4

Question

I am trying to capture an RTSP stream from a VIRB 360 camera, into OpenCV . The video is H264 and according to one of the comments here , OpenCV 3.4 should be able to handle it. Here is the code:

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/videoio.hpp>

int main()
{
    cv::VideoCapture cap("rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720&liveStreamActive=1", cv::CAP_FFMPEG);

    if(!cap.isOpened())
    {   
        std::cout << "Input error\n";
        return -1;
    }

    cv::namedWindow("Video Feed", cv::WINDOW_AUTOSIZE);

    cv::Mat frame;
    for(;;)
    {
        //std::cout << "Format: " << cap.get(CV_CAP_PROP_FORMAT) << "\n";
        cap >> frame;
        cv::imshow("Video Feed", frame);    
        if (cv::waitKey(10) == 27)
        {
            break;
        }
    }   
    cv::destroyAllWindows();
    return 0;
}

I have compiled OpenCV with ffmpeg and gstreamer capabilities. When I run the following Gstreamer command, I am able to stream it, but with a delay of 3 seconds (not acceptable):

 gst-launch-1.0 playbin uri=rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720\&liveStreamActive=1

On the other hand, I get a 0.5 second delay using ffplay/ffmpeg command (acceptable):

ffplay -fflags nobuffer -rtsp_transport udp rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720\&liveStreamActive=1

or

ffplay -probesize 32 -sync ext rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720\&liveStreamActive=1

In the OpenCV code written above, using cv::CAP_FFMPEG flag in the line:

cv::VideoCapture cap("rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720&liveStreamActive=1", cv::CAP_FFMPEG);

gives the error:

[rtsp @ 0x2312040] The profile-level-id field size is invalid (65)
[rtsp @ 0x2312040] method SETUP failed: 461 Unsupported transport
Input error

If I use cv::CAP_GSTREAMER , it throws no error, but nothing happens. I believe that the problem is that OpenCV is not able to handle UDP transport layer. What are the possible solutions? Kindly provide suggestions.

Edit 1: I was able to get capture the stream by following this . I made the following changes: instead of cv::VideoCapture cap("rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720&liveStreamActive=1", cv::CAP_FFMPEG); the code now has:

#if WIN32
    _putenv_s("OPENCV_FFMPEG_CAPTURE_OPTIONS", "rtsp_transport;udp");
#else
    setenv("OPENCV_FFMPEG_CAPTURE_OPTIONS", "rtsp_transport;udp", 1);
#endif
auto cap = cv::VideoCapture("rtsp://192.168.0.1/livePreviewStream?maxResolutionVertical=720&liveStreamActive=1", cv::CAP_FFMPEG);

#if WIN32
    _putenv_s("OPENCV_FFMPEG_CAPTURE_OPTIONS", "");
#else
    unsetenv("OPENCV_FFMPEG_CAPTURE_OPTIONS");
#endif

However, it throws the following errors:

[rtsp @ 0x2090580] The profile-level-id field size is invalid (65)
[rtsp @ 0x2090580] Error parsing AU headers
[h264 @ 0x208d240] error while decoding MB 69 40, bytestream -7
[rtsp @ 0x2090580] Error parsing AU headers
[rtsp @ 0x2090580] Error parsing AU headers
[h264 @ 0x2316700] left block unavailable for requested intra4x4 mode -1
[h264 @ 0x2316700] error while decoding MB 0 16, bytestream 112500
[rtsp @ 0x2090580] Error parsing AU headers

which means the video is sometimes glitchy and looks like:

I believe it has something to do with:

setenv("OPENCV_FFMPEG_CAPTURE_OPTIONS", "rtsp_transport;udp", 1);

I would appreciate any suggestions or improvements. Thank You.

Answer 1

Set this:

cv::VideoCapture cap;
cap.set(CV_CAP_PROP_BUFFERSIZE, 3); /

I think this was answered here. OpenCV VideoCapture lag due to the capture buffer