OpenCVSharp - Compare images - Sift/Surf/Hash/Orb

Question

I know the subject was widely discussed over in the internet however I couldn't find an answer to my question...

So the ultimate goal of mine is to extract an part of image and compare it with "Template". I've already extracted and cropped the part of image so that's not the problem.

The issue I am having is to trying to compare it to it's template. Template was originally in SVG so I converted it to JPG and removed alpha channel.

Template:

Actual image is take from webcam so It's coloured but I can easily do threshold to leave only black/white colour.

Webcam image:

Webcam image with threshold

So I tried 3 methods.

1. SIFT algorithm. 
 

private void MatchBySift(Mat src1, Mat src2)

        {

            var gray1 = new Mat();
            var gray2 = new Mat();

            Cv2.CvtColor(src1, gray1, ColorConversionCodes.BGR2GRAY);
            Cv2.CvtColor(src2, gray2, ColorConversionCodes.BGR2GRAY);

            var sift = SIFT.Create();

            // Detect the keypoints and generate their descriptors using SIFT
            KeyPoint[] keypoints1, keypoints2;
            var descriptors1 = new Mat();   //<float>
            var descriptors2 = new Mat();   //<float>
            sift.DetectAndCompute(gray1, null, out keypoints1, descriptors1);
            sift.DetectAndCompute(gray2, null, out keypoints2, descriptors2);

            // Match descriptor vectors
            var bfMatcher = new BFMatcher(NormTypes.L2, false);
            var flannMatcher = new FlannBasedMatcher();
            DMatch[] bfMatches;
            DMatch[] flannMatches;
            try
            {
                bfMatches = bfMatcher.Match(descriptors1, descriptors2);
                flannMatches = flannMatcher.Match(descriptors1, descriptors2);
            }
            catch
            {
                bfMatches = new DMatch[0];
                flannMatches = new DMatch[0];
            }
            

            // Draw matches
            var bfView = new Mat();
            Cv2.DrawMatches(gray1, keypoints1, gray2, keypoints2, bfMatches, bfView);
            var flannView = new Mat();
            Cv2.DrawMatches(gray1, keypoints1, gray2, keypoints2, flannMatches, flannView);
            Console.WriteLine("BF Matches: " + bfMatches.Length);
            Console.WriteLine("Flann Matches: " + flannMatches.Length);
            
            using (new Window("SIFT matching (by BFMather)", bfView))
            using (new Window("SIFT matching (by FlannBasedMatcher)", flannView))
            {
                Cv2.WaitKey();
            }
            
        }

2. SURF Algorithm

    private void MatchBySurf(Mat src1, Mat src2)
            {
                var gray1 = new Mat();
                var gray2 = new Mat();
    
                Cv2.CvtColor(src1, gray1, ColorConversionCodes.BGR2GRAY);
                Cv2.CvtColor(src2, gray2, ColorConversionCodes.BGR2GRAY);
    
                var surf = SURF.Create(200, 4, 2, true);
    
                // Detect the keypoints and generate their descriptors using SURF
                KeyPoint[] keypoints1, keypoints2;
                var descriptors1 = new Mat();   //<float>
                var descriptors2 = new Mat();   //<float>
                surf.DetectAndCompute(gray1, null, out keypoints1, descriptors1);
                surf.DetectAndCompute(gray2, null, out keypoints2, descriptors2);
    
                // Match descriptor vectors 
                var bfMatcher = new BFMatcher(NormTypes.L2, false);
                var flannMatcher = new FlannBasedMatcher();
                DMatch[] bfMatches;
                DMatch[] flannMatches;
                try
                {
                    bfMatches = bfMatcher.Match(descriptors1, descriptors2);
                    flannMatches = flannMatcher.Match(descriptors1, descriptors2);
                }
                catch
                {
                    bfMatches = new DMatch[0];
                    flannMatches = new DMatch[0];
                }
    
                // Draw matches
                var bfView = new Mat();
                Cv2.DrawMatches(gray1, keypoints1, gray2, keypoints2, bfMatches, bfView);
                var flannView = new Mat();
                Cv2.DrawMatches(gray1, keypoints1, gray2, keypoints2, flannMatches, flannView);
    
                Console.WriteLine("BF Matches: " + bfMatches.Length);
                Console.WriteLine("Flann Matches: " + flannMatches.Length);
                /*
                using (new Window("SURF matching (by BFMather)", bfView))
                using (new Window("SURF matching (by FlannBasedMatcher)", flannView))
                {
                    Cv2.WaitKey();
                }
                */
            }

3. Hash comparison

    private void MatchByHash(Mat src1, Mat src2)
            {
                Bitmap bit1 = BitmapConverter.ToBitmap(src1);
                Bitmap bit2 = BitmapConverter.ToBitmap(src2);
                List<bool> hash1 = GetHash(bit1);
                List<bool> hash2 = GetHash(bit2);
    
                int equalElements = hash1.Zip(hash2, (i, j) => i == j).Count(eq => eq);
    
                Console.WriteLine("Match by Hash: " + equalElements);
            }
    
            public List<bool> GetHash(Bitmap bmpSource)
            {
                List<bool> lResult = new List<bool>();
                //create new image with 16x16 pixel
                Bitmap bmpMin = new Bitmap(bmpSource, new System.Drawing.Size(16, 16));
                for (int j = 0; j < bmpMin.Height; j++)
                {
                    for (int i = 0; i < bmpMin.Width; i++)
                    {
                        //reduce colors to true / false                
                        lResult.Add(bmpMin.GetPixel(i, j).GetBrightness() < 0.5f);
                    }
                }
                return lResult;
            }

Why I am not happy with results

1. SIFT and SURF

When comparing above images I get certain similarity based on number of matches. Problem is when I use different template image I get the same numbers all the time... I think it's because there is no colour but I need clarification here

2. Hash

When used on other 'Template' I got higher result. Which basically eliminates this approach unless it could be improved somehow...

I also heard of ORB algorithm but I've not tried to implement it yet. Would that work better on my occasion?

If there is any other approach I could try, please point me in the right direction!

Thank You

Answer 1

The main thing is to register the logo and the template. As your images are quasi binary and contain only the logo, you can rely on the computation of moments. You will compute them for the binarized image and the binarized template.

The zeroth order moment of the black pixels is the area. The square root of the ratio of the areas gives you the scaling factor.

The first order X at Y moments give you the position of a reference point (the centroid), which will be the same in both.

Then the centered second order moments (X², Y², XY) will give you two main directions (by computing the Eigenvectors of the inertia tensor), and from this, the rotation angle between the logo and the template.

https://www.ae.msstate.edu/vlsm/shape/area_moments_of_inertia/papmi.htm

Now you can rescale, rotate and translate either the logo or the template to superimpose them. You will obtain a comparison score by simply counting the pixels of matching color.