使用EmguCV 3.1.0.1从视频捕获中检测

Question

I'm trying to implement face detection with video capture using EmguCV 3.1.0.1 library, installed by NuGet packages of VS15 on PC Windows 10 64-bit OS in WinForms desktop application.

My goal is to get detection and tracking of human face from videocamera and detect smile, but for examples below I will use only face HaarCascade .xml with CascadeClassifier .

So, my using with DirectShowLib library for videoDevice from comboBox1_SelectedIndexChanged SelectedItem :

using Emgu.CV;
using Emgu.CV.Structure;
using Emgu.CV.UI;
using DirectShowLib;

Path to HaarCascade xml-s:

    string facePath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData), "dir\\haarcascade_frontalface_default.xml");

Timer:

    private void timer1_Tick(object sender, EventArgs e)
    {
        detectFace();
    }

Attempt 1:

    private void detectFace()
    {
        CascadeClassifier face = new CascadeClassifier(facePath);
        Image<Bgr, Byte> currentframe = null;
        Image<Gray, byte> grayFrame = null;
        Capture grabber;
        grabber = new Capture(videoDevice);
        currentframe = grabber.QueryFrame().Resize(500, 320, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC);
        if (currentframe != null)
        {
            grayFrame = currentframe.Convert<Gray, Byte>();
            Rectangle[] faceDetected = face.DetectMultiScale(grayFrame, 1.1, 10, Size.Empty, Size.Empty);
            foreach (Rectangle faceFound in faceDetected)
            {
                currentframe.Draw(faceFound, new Bgr(Color.Red), 2);
            }
            pictureBox1.Image = currentframe.ToBitmap();
        }
    }

line currentframe = grabber.QueryFrame().Resize(500, 320, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC); says:

Error CS0234 The type or namespace name 'INTER' does not exist in the namespace 'Emgu.CV.CvEnum' (are you missing an assembly reference?)

Instead I've tried to use currentframe = grabber.QueryFrame().Resize(500, 320, Emgu.CV.CvEnum.TemplateMatchingType.CcoeffNormed); , with grabber.QueryFrame().MatchTemplate or grabber.QueryFrame().Retrieve , but another error still keeps in same line as:

Error CS1061 'Mat' does not contain a definition for 'Resize' and no extension method 'Resize' accepting a first argument of type 'Mat' could be found (are you missing a using directive or an assembly reference?)

I'm not sure where I have to download required dll-s (if it is a reason of missing?) and which dll-s exactly I should add to references additionally.

Attempt 2:

    private Capture _capture;
    private CascadeClassifier _cascadeClassifier;
    private void detectFace()
    {
        _capture = new Capture(videoDevice); 
        _cascadeClassifier = new CascadeClassifier(facePath);
        using (var imageFrame = _capture.QueryFrame().ToImage())
        {
            if (imageFrame != null)
            {
                var grayframe = imageFrame.Convert();
                var faces = _cascadeClassifier.DetectMultiScale(grayframe, 1.1, 10, Size.Empty);
                foreach (var face in faces)
                {
                    imageFrame.Draw(face, new Bgr(Color.Red), 3);
                }
            }
            pictureBox1.Image = imageFrame.ToBitmap();
        }
    }

line using (var imageFrame = _capture.QueryFrame().ToImage()) :

Error CS0411 The type arguments for method 'Mat.ToImage(bool)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

line var grayframe = imageFrame.Convert(); :

Error CS0411 The type arguments for method 'Image.Convert()' cannot be inferred from the usage. Try specifying the type arguments explicitly.

line imageFrame.Draw(face, new Bgr(Color.Red), 3); :

Error CS1503 Argument 2: cannot convert from 'Emgu.CV.Structure.Bgr' to 'TColor'

Any guide, advice or example would be helpful

Edit by Michal Nawrocik answer below:

Method 1:

private void detectFace()
        {
            CascadeClassifier face = new CascadeClassifier(facePath);
            Image<Bgr, Byte> currentframe = null;
            Image<Gray, byte> grayFrame = null;
            Capture grabber;
            grabber = new Capture(videoDevice);  
            var dstMat = new Mat();
            var frame = grabber.QueryFrame();
            CvInvoke.Resize(frame, dstMat, new Size(500, 320), interpolation: Emgu.CV.CvEnum.Inter.Cubic);
            currentframe = dstMat.ToImage<Bgr, byte>();
            if (currentframe != null)
            {
                grayFrame = currentframe.Convert<Gray, Byte>();
                Rectangle[] faceDetected = face.DetectMultiScale(grayFrame, 1.1, 10, Size.Empty, Size.Empty);
                foreach (Rectangle faceFound in faceDetected)
                {
                    currentframe.Draw(faceFound, new Bgr(Color.Red), 2);
                }
                pictureBox1.Image = currentframe.ToBitmap();
            }
        }

Unhandled exception:

System.AccessViolationException' occurred in Emgu.CV.World.dll Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt

Method 2:

 private Capture _capture;
    private CascadeClassifier _cascadeClassifier;
    private void detectFace()
    {
            _capture = new Capture(videoDevice);   
            _cascadeClassifier = new CascadeClassifier(facePath);
            using (var imageFrame = _capture.QueryFrame().ToImage<Bgr, byte>())
            {
                if (imageFrame != null)
                {
                    var grayframe = imageFrame.Convert<Gray, byte>();
                    var faces = _cascadeClassifier.DetectMultiScale(grayframe, 1.1, 10, Size.Empty);  
                    foreach (var face in faces)
                    {
                        imageFrame.Draw(face, new Bgr(Color.Red), 3);
                    }
                }
                pictureBox1.Image = imageFrame.ToBitmap();
            }
        }           
    }

exception:

Emgu.CV.Util.CvException' occurred in Emgu.CV.World.dll Additional information: OpenCV: Unrecognized or unsupported array type

Answer 1

As your error message says, Mat class does not have a Resize() instance method, some quick googling reveals that you need to call the static method of CvInvoke class. The enum value for cubic interpolation has also been changed by developers of EmguCV at some point in time.

Here is the working code corresponding to your attempt 1:

var dstMat = new Mat();
var frame = grabber.QueryFrame();
CvInvoke.Resize(frame, dstMat, new Size(500, 320), interpolation: Emgu.CV.CvEnum.Inter.Cubic);
currentframe = dstMat.ToImage<Bgr, byte>();

Notice that I save references to Mat objects in order to dispose them later. using statements can also be used instead.

Your other problems are similar. There are no parameterless non-generic methods like Mat.ToImage() or Image.Convert() . They would not even make much sense because you need to specify what format you want to convert to. In your case you can use:

var imageFrame =_capture.QueryFrame().ToImage<Bgr, byte>();
var grayFrame = imageFrame.Convert<Gray, byte>();

The last error in Image.Draw() call will disappear then.

Your problems suggest that you might benefit from taking a closer look at classes metadata available eg via Object Browser built in Visual Studio.

Edit:

In the first method you forgot to call Dispose() for some objects and this caused the exception. Here is the corrected code, tested and working:

private void detectFace()
{
    CascadeClassifier face = new CascadeClassifier(facePath);
    Image<Bgr, Byte> currentframe = null;
    Image<Gray, byte> grayFrame = null;
    Capture grabber;
    grabber = new Capture(videoDevice);
    var dstMat = new Mat();
    var frame = grabber.QueryFrame();
    CvInvoke.Resize(frame, dstMat, new Size(500, 320), interpolation: Emgu.CV.CvEnum.Inter.Cubic);
    currentframe = dstMat.ToImage<Bgr, byte>();
    if (currentframe != null)
    {
        grayFrame = currentframe.Convert<Gray, Byte>();
        Rectangle[] faceDetected = face.DetectMultiScale(grayFrame, 1.1, 10, Size.Empty, Size.Empty);
        foreach (Rectangle faceFound in faceDetected)
        {
            currentframe.Draw(faceFound, new Bgr(Color.Red), 2);
        }

        var oldImage = panAndZoomPictureBox1.Image;
        panAndZoomPictureBox1.Image = currentframe.ToBitmap();
        if (oldImage != null)
        {
            oldImage.Dispose();
        }

        currentframe.Dispose();
        grayFrame.Dispose();
    }

    face.Dispose();

    grabber.Dispose();

    dstMat.Dispose();
    frame.Dispose();
}

I made the least possible fixes to make it work. There is quite some room for improvement. You can create Capture and CascadeClassifier just once instead of each timer tick, this will improve performance considerably.

I also noticed that the version of Emgu package you are using is quite old even though NuGet does not show newer version for this very package. The reason is it's been renamed to EMGU.CV.