How to minimize the lag from the live stream using OpenCV/EmguCV WPF

Question

I am using EmguCV to get a live stream from a high-resolution webcam. (full HD) The issue is that there is a significant lag in the video. I compared with the Windows camera app and my application is much more delayed.

Here is the code snippet I am using to get the live stream and show on the canvas. I am wondering if I am missing anything important to minimize the lag. If anybody is experienced, please help me.

void init_camera()
{
    m_capture= new VideoCapture(0);
    m_capture.ImageGrabbed += ProcessFrame;
}

void ProcessFrame()
{
    if (m_capture_in_progress)
        return;
    m_capture_in_progress = true;
    if (m_capture != null && m_capture.Ptr != IntPtr.Zero)
    {
        m_capture.Retrieve(m_frame, 0);
        if (m_frame != null && m_frame.Bitmap != null)
        {
            if (IsRecording && m_video_writer != null && m_video_writer.IsOpened)
            {
                try
                {
                    m_video_writer.Write(m_frame);
                }
                catch (Exception ex)
                {
                    log(ex.Message + "\n" + ex.StackTrace);
                }
            }

            this.Dispatcher.Invoke(() =>
            {
                using (var stream = new MemoryStream())
                {
                    m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
                    using (var outstream = new MemoryStream(stream.ToArray()))
                    {

                        BitmapImage bitmap = new BitmapImage();
                        bitmap.BeginInit();
                        bitmap.StreamSource = new MemoryStream(stream.ToArray());
                        bitmap.CacheOption = BitmapCacheOption.OnLoad;
                        bitmap.EndInit();

                        ui_canvas.Background = new ImageBrush(bitmap);
                    }
                };
            });
        }
    }
    m_capture_in_progress = false;
}

Answer 1

OP is referring some kind of laggs while drawing a camera capture stream from EmguCv into canvas wpf control. OP also states that lagg disappears when streaming with third party software.

This is caused by low performance in OP's code.

After wataching the posted code sample I would suggest the following improvements:

1. Increasing FPS rate from EmguCV parameters

Maybe your device is supporting higher fps rates, try to increase this value.

2. Drawing the bitmap over another control like image

As far as I know, drawing a bitmap over a canvas is slower than drawing it over an image control. A good solution would be to overlap both controls so you can paint the frames over the image and draw your shapes over the canvas as they are well overlapped.

3. Cleaning up the rendering method

Keep simple and try to avoid so much control structures when you are inside a critical program block. Specially in program blocks that are executed when an external devices fires an event.

4. Fixing possible program lock

As far as I can see, you are locking the ProcessFrame event with a bool variable called m_capture_in_progress which is set to true while you are drawing the frame and it is freed after you finish the drawing. This can cause that next incoming frames are not painted because the method is still blocked by the previous frame. This can cause a low performance issue as many frames get lost and not painted.

Answer 2

Use the Image control but set its Source Property to a WriteableBitmap instead of a BitmapImage. Then lock the bitmap, copy the pixel data from your EmguCV Mat to the backbuffer of the WriteableBitmap, call AddDirtyRect method and finally unlock the Bitmap. The final unlock call in combination with the AddDirtyRect will trigger a redraw of the UI image.

Adavantages:

• It does not generate a new BitmapImage object each time you want to draw a new frame.
• You copy the pixel data only once

Your copy, encode and decode data to often:

 // Copies and encodes pixel data to jpeg stream
 m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);

 // Copies jpeg stream
 using (var outstream = new MemoryStream(stream.ToArray()))
 {

     BitmapImage bitmap = new BitmapImage();
     bitmap.BeginInit();

     //Copies jpeg stream again
     bitmap.StreamSource = new MemoryStream(stream.ToArray());

     bitmap.CacheOption = BitmapCacheOption.OnLoad;

     // Triggers jpeg stream decoding
     bitmap.EndInit();
     ui_canvas.Background = new ImageBrush(bitmap);
  }