Using opencv to find the most similar image that contains another image

Question

If the title isn't clear let's say I have a list of images (10k+), and I have a target image I am searching for.

Here's an example of the target image:

Here's an example of images I will want to be searching to find something 'similar' (ex1, ex2, and ex3):

Here's the matching I do (I use KAZE)

from matplotlib import pyplot as plt
import numpy as np
import cv2
from typing import List
import os
import imutils


def calculate_matches(des1: List[cv2.KeyPoint], des2: List[cv2.KeyPoint]):
    """
    does a matching algorithm to match if keypoints 1 and 2 are similar
    @param des1: a numpy array of floats that are the descriptors of the keypoints
    @param des2: a numpy array of floats that are the descriptors of the keypoints
    @return:
    """
    # bf matcher with default params
    bf = cv2.BFMatcher(cv2.NORM_L2)
    matches = bf.knnMatch(des1, des2, k=2)
    topResults = []
    for m, n in matches:
        if m.distance < 0.7 * n.distance:
            topResults.append([m])

    return topResults


def compare_images_kaze():
    cwd = os.getcwd()
    target = os.path.join(cwd, 'opencv_target', 'target.png')
    images_list = os.listdir('opencv_images')
    for image in images_list:
        # get my 2 images
        img2 = cv2.imread(target)
        img1 = cv2.imread(os.path.join(cwd, 'opencv_images', image))
        for i in range(0, 360, int(360 / 8)):
            # rotate my image by i
            img_target_rotation = imutils.rotate_bound(img2, i)

            # Initiate KAZE object with default values
            kaze = cv2.KAZE_create()
            kp1, des1 = kaze.detectAndCompute(img1, None)
            kp2, des2 = kaze.detectAndCompute(img2, None)
            matches = calculate_matches(des1, des2)

            try:
                score = 100 * (len(matches) / min(len(kp1), len(kp2)))
            except ZeroDivisionError:
                score = 0
            print(image, score)
            img3 = cv2.drawMatchesKnn(img1, kp1, img_target_rotation, kp2, matches,
                                      None, flags=2)
            img3 = cv2.cvtColor(img3, cv2.COLOR_BGR2RGB)
            plt.imshow(img3)
            plt.show()
            plt.clf()


if __name__ == '__main__':
    compare_images_kaze()

Here's the result of my code:

ex1.png 21.052631578947366
ex2.png 0.0
ex3.png 42.10526315789473

It does alright, It was able to tell that ex1 is similar and ex2 is not similar. however it states that ex3 is similar (even more similar than ex1), Any extra pre-processing or post-processing (maybe ml? assuming ml is actually useful) or just changes I can do to my method that can be done to keep only ex1 as similar and not ex3?

(Note this score I create is something I found online. Not sure if it's an accurate way to go about it)

ADDED MORE EXAMPLES BELOW

Another set of examples:

Here's what I am searching for

I want the above image to be similar to the middle and bottom images (NOTE: I rotate my target image by 45 degrees and compare it to the images below.)

Feature matching (as stated in answers below) were useful in found similarity with the second image, but not the third image (Even after rotating it properly)

Answer 1

Detecting The Most Similar Image

The Code

You can use template matching , where the image you want to detect if it's in the other images is the template. I have that small image saved in template.png , and the other three images in img1.png , img2.png and img3.png .

I defined a function that utilizes the cv2.matchTemplate to calculate the amount of confidence for if a template is in an image. Using the function on every image, the one that results ion the highest confidence is the image that contains the template:

import cv2

template = cv2.imread("template.png", 0)
files = ["img1.png", "img2.png", "img3.png"]

for name in files:
    img = cv2.imread(name, 0)
    print(f"Confidence for {name}:")
    print(cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED).max())

The Output:

Confidence for img1.png:
0.8906427
Confidence for img2.png:
0.4427919
Confidence for img3.png:
0.5933967

The Explanation:

1. Import the opencv module, and read in the template image as grayscale by setting the second parameter of the cv2.imread method to 0 :

import cv2

template = cv2.imread("template.png", 0)

2. Define your list of images of which you want to determine which one contains the template:

files = ["img1.png", "img2.png", "img3.png"]

3. Loop through the filenames and read in each one as a grayscale image:

for name in files:
    img = cv2.imread(name, 0)

4. Finally, you can use the cv2.matchTemplate to detect the template in each image. There are many detection methods you can use, but for this I decided to use the cv2.TM_CCOEFF_NORMED method:

    print(f"Confidence for {name}:")
    print(cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED).max())

The output of the function ranges from between 0 and 1 , and as you can see, it successfully detected that the first image is most likely to contain the template image (it has the highest level of confidence) .

---

The Visualization

The Code

If detecting which image contains the template isn't enough, and you want a visualization, you can try the code below:

import cv2
import numpy as np

def confidence(img, template):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
    res = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)
    conf = res.max()
    return np.where(res == conf), conf

files = ["img1.png", "img2.png", "img3.png"]

template = cv2.imread("template.png")
h, w, _ = template.shape

for name in files:
    img = cv2.imread(name)
    ([y], [x]), conf = confidence(img, template)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    text = f'Confidence: {round(float(conf), 2)}'
    cv2.putText(img, text, (x, y), 1, cv2.FONT_HERSHEY_PLAIN, (0, 0, 0), 2)
    cv2.imshow(name, img)
    
cv2.imshow('Template', template)
cv2.waitKey(0)

The Output:

The Explanation:

1. Import the necessary libraries:

import cv2
import numpy as np

2. Define a function that will take in a full image and a template image. As the cv2.matchTemplate method requires grayscale images, convert the 2 images into grayscale:

def confidence(img, template):
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)

3. Use the cv2.matchTemplate method to detect the template in the image, and return the position of the point with the highest confidence, and return the highest confidence:

    res = cv2.matchTemplate(img, template, cv2.TM_CCOEFF_NORMED)
    conf = res.max()
    return np.where(res == conf), conf

4. Define your list of images you want to determine which one contains the template, and read in the template image:

files = ["img1.png", "img2.png", "img3.png"]
template = cv2.imread("template.png")

5. Get the size of the template image to later use for drawing a rectangle on the images:

h, w, _ = template.shape

6. Loop though the filenames and read in each image. Using the confidence function we defined before, get the xy position of the top-left corner of the detected template and the confidence amount for the detection:

for name in files:
    img = cv2.imread(name)
    ([y], [x]), conf = confidence(img, template)

7. Draw a rectangle on the image at the corner and put the text on the image. Finally, show the image:

    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    text = f'Confidence: {round(float(conf), 2)}'
    cv2.putText(img, text, (x, y), 1, cv2.FONT_HERSHEY_PLAIN, (0, 0, 0), 2)
    cv2.imshow(name, img)

8. Also, show the template for comparison:

cv2.imshow('Template', template)
cv2.waitKey(0)

Answer 2

I'm not sure, if the given images resemble your actual task or data, but for this kind of images, you could try simple template matching, cf. this OpenCV tutorial .

Basically, I just implemented the tutorial with some modifications:

import cv2
import matplotlib.pyplot as plt

# Read images
examples = [cv2.imread(img) for img in ['ex1.png', 'ex2.png', 'ex3.png']]
target = cv2.imread('target.png')
h, w = target.shape[:2]

# Iterate examples
for i, img in enumerate(examples):

    # Template matching
    # cf. https://docs.opencv.org/4.5.2/d4/dc6/tutorial_py_template_matching.html
    res = cv2.matchTemplate(img, target, cv2.TM_CCOEFF_NORMED)

    # Get location of maximum
    _, max_val, _, top_left = cv2.minMaxLoc(res)

    # Set up threshold for decision target found or not
    thr = 0.7
    if max_val > thr:

        # Show found target in example
        bottom_right = (top_left[0] + w, top_left[1] + h)
        cv2.rectangle(img, top_left, bottom_right, (0, 255, 0), 2)

    # Visualization
    plt.figure(i, figsize=(10, 5))
    plt.subplot(1, 2, 1), plt.imshow(img[..., [2, 1, 0]]), plt.title('Example')
    plt.subplot(1, 2, 2), plt.imshow(res, vmin=0, vmax=1, cmap='gray')
    plt.title('Matching result'), plt.colorbar(), plt.tight_layout()

plt.show()

These are the results:

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.16299-SP0
Python:        3.9.1
PyCharm:       2021.1.1
Matplotlib:    3.4.1
OpenCV:        4.5.1
----------------------------------------

---

EDIT: To emphasize the information from the different colors, one might use the hue channel from the HSV color space for the template matching:

import cv2
import matplotlib.pyplot as plt

# Read images
examples = [
    [cv2.imread(img) for img in ['ex1.png', 'ex2.png', 'ex3.png']],
    [cv2.imread(img) for img in ['ex12.png', 'ex22.png', 'ex32.png']]
]
targets = [
    cv2.imread('target.png'),
    cv2.imread('target2.png')
]

# Iterate examples and targets
for i, (ex, target) in enumerate(zip(examples, targets)):
    for j, img in enumerate(ex):

        # Rotate last image from second data set
        if (i == 1) and (j == 2):
            img = cv2.rotate(img, cv2.ROTATE_90_CLOCKWISE)

        h, w = target.shape[:2]

        # Get hue channel from HSV color space
        target_h = cv2.cvtColor(target, cv2.COLOR_BGR2HSV)[..., 0]
        img_h = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)[..., 0]

        # Template matching
        # cf. https://docs.opencv.org/4.5.2/d4/dc6/tutorial_py_template_matching.html
        res = cv2.matchTemplate(img_h, target_h, cv2.TM_CCOEFF_NORMED)

        # Get location of maximum
        _, max_val, _, top_left = cv2.minMaxLoc(res)

        # Set up threshold for decision target found or not
        thr = 0.6
        if max_val > thr:

            # Show found target in example
            bottom_right = (top_left[0] + w, top_left[1] + h)
            cv2.rectangle(img, top_left, bottom_right, (0, 255, 0), 2)

        # Visualization
        plt.figure(i * 10 + j, figsize=(10, 5))
        plt.subplot(1, 2, 1), plt.imshow(img[..., [2, 1, 0]]), plt.title('Example')
        plt.subplot(1, 2, 2), plt.imshow(res, vmin=0, vmax=1, cmap='gray')
        plt.title('Matching result'), plt.colorbar(), plt.tight_layout()
        plt.savefig('{}.png'.format(i * 10 + j))

plt.show()

New results:

Answer 3

The Concept

We can use the cv2.matchTemplate method to detect where an image is in another image, but for your second set of images you have rotation. Also, we'll need to take the colors into account.

cv2.matchTemplate will take in an image, a template (the other image) and a template detection method, and will return a grayscale array where the brightest point in the grayscale array will be the point with the most confidence that template is at that point.

We can use the template at 4 different angles and use the one that resulted in the highest confidence. When we detected a possible point that matched the template, we use a function (that we will define ourselves) to check if the most frequent colors in the template is present in the patch of the image we detected. If not, then ignore the patch, regardless of the amount of confidence returned.

The Code

import cv2
import numpy as np

def frequent_colors(img, vals=3):
    colors, count = np.unique(np.vstack(img), return_counts=True, axis=0)
    sorted_by_freq = colors[np.argsort(count)]
    return sorted_by_freq[-vals:]

def get_templates(img):
    template = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    for i in range(3):
        yield cv2.rotate(template, i)
        
def detect(img, template, min_conf=0.45):
    colors = frequent_colors(template)
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    conf_max = min_conf
    shape = 0, 0, 0, 0
    for tmp in get_templates(template):
        h, w = tmp.shape
        res = cv2.matchTemplate(img_gray, tmp, cv2.TM_CCOEFF_NORMED)
        for y, x in zip(*np.where(res > conf_max)):
            conf = res[y, x]
            if conf > conf_max:
                seg = img[y:y + h, x:x + w]
                if all(np.any(np.all(seg == color, -1)) for color in colors):
                    conf_max = conf
                    shape = x, y, w, h
    return shape

files = ["img1_2.png", "img2_2.png", "img3_2.png"]
template = cv2.imread("template2.png")

for name in files:
    img = cv2.imread(name)
    x, y, w, h = detect(img, template)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    cv2.imshow(name, img)

cv2.imshow('Template', template)
cv2.waitKey(0)

The Output

The Explanation

1. Import the necessary libraries:

import cv2
import numpy as np

2. Define a function, frequent_colors , that will take in an image and return the most frequent colors in the image. An optional parameter, val , is how many colors to return; if val is 3 , then the 3 most frequent colors will be returned:

def frequent_colors(img, vals=3):
    colors, count = np.unique(np.vstack(img), return_counts=True, axis=0)
    sorted_by_freq = colors[np.argsort(count)]
    return sorted_by_freq[-vals:]

3. Define a function, get_templates , that will take in an image, and yield the image (in grayscale) at 4 different angles - original, 90 clockwise, 180, and 90 counterclockwise:

def get_templates(img):
    template = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    for i in range(3):
        yield cv2.rotate(template, i)

4. Define a function, detect , that will take in an image and a template image, and return the x, y, w, h of the bounding box of the detected template on the image, and for this function we will be utilizing the frequent_colors and get_templates functions defined earlier. The min_conf parameter will be the minimum amount of confidence needed to classify a detection as an actual detection:

def detect(img, template, min_conf=0.45):

5. Detect the three most frequent colors in the template and store them in a variable, colors . Also, define a grayscale version of the main image:

    colors = frequent_colors(template)
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

6. Define the initial value for the greatest confidence detected, and initial values for the detected patch:

    conf_max = min_conf
    shape = 0, 0, 0, 0

7. Loop though the grayscale templates at 4 angles, get the shape of the grayscale template (as rotation changes the shape) , and use the cv2.matchTemplate method to get the grayscale array of detected templates on the image:

    for tmp in get_templates(template):
        h, w = tmp.shape
        res = cv2.matchTemplate(img_gray, tmp, cv2.TM_CCOEFF_NORMED)

8. Loop though the x, y coordinates of the detected templates where the confidence is greater than conf_min , and store the confidence in a variable, conf . If conf is greater than the initial greatest confidence variable ( conf_max ) , proceed to detect if all three most frequent colors in the template is present in the patch of the image:

        for y, x in zip(*np.where(res > conf_max)):
            conf = res[y, x]
            if conf > conf_max:
                seg = img[y:y + h, x:x + w]
                if all(np.any(np.all(seg == color, -1)) for color in colors):
                    conf_max = conf
                    shape = x, y, w, h

9. At the end we can return the shape. If no template is detected in the image, the shape will be the initial values defined for it, 0, 0, 0, 0 :

    return shape

10. Finally, loop though each image and use the detect function we defined to get the x, y, w, h of the bounding box. Use the cv2.rectangle method to draw the bounding box onto the images:

files = ["img1_2.png", "img2_2.png", "img3_2.png"]
template = cv2.imread("template2.png")

for name in files:
    img = cv2.imread(name)
    x, y, w, h = detect(img, template)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 2)
    cv2.imshow(name, img)

cv2.imshow('Template', template)
cv2.waitKey(0)

Answer 4

First, the data appears in graphs, aren't you able to get the overlapping values from their numerical data?

And have you tried performing some edge detection for the change in color from white-blue and then from blue-red, fitting some circles to those edges and then checking if they overlap?

Since the input data is quite controlled (no organic photography or videos), perhaps you won't have to go the ML route.