Use matchTemplate with colored images OpenCV

Question

I am trying to detect certain COLORED images in a desktop screenshot, where I have equally shaped templates but different color (these are not distinguished from one another doing the normal matchTemplate method as it is done with greyscale images) here is the main code that does the detection:

     template = cv2.imread(template_path,1)
    #template_hsv = cv2.cvtColor(template, cv2.COLOR_RGB2HSV)
    #template_B, template_G, template_R = cv2.split(template)
    #scr_B, scr_G, scr_R = cv2.split(screenshot)
    scr_B = screenshot[:, :, 0]
    scr_G = screenshot[:, :, 1]
    scr_R = screenshot[:, :, 2]
    template_B = template[:, :, 0]
    template_G = template[:, :, 1]
    template_R = template[:, :, 2]
    #cv2.imwrite('./sc4.png', scr_R)
    #cv2.imwrite('./template.png', template)

    resB = cv2.matchTemplate(scr_B, template_B, cv2.TM_CCOEFF_NORMED)
    resG = cv2.matchTemplate(scr_G, template_G, cv2.TM_CCOEFF_NORMED)
    resR = cv2.matchTemplate(scr_R, template_R, cv2.TM_CCOEFF_NORMED)

    res = resB + resG + resR

    #res = cv2.matchTemplate(screenshot, template_G, cv2.TM_CCOEFF_NORMED)
    matches = np.where(res >= 3*threshold)
    print(matches)
    return matches

As you can see I tried splitting the channels of the rgb screenshot image, and compare then with the also splitted template image. I also tried doing this with HSV channels as you can see in the commented code. However this did not work and despite seeing that in single channeled images there was a visual difference of color, the program did not distinguish them (I also tried doing comparisons with each individual channel of the template and screenshot) .

All sugestions are welcome, even trying to achieve my goal using anything else. Thank you in advance.

Answer 1

I tried it with TM_CCOEFF_NORMED method and did not work... somehow it gave everything as 1.0 (all max value).... but with the TM_SQDIFF_NORMED method actually gives something reasonable.

Lets start by creating a sample image, I did it as following:

import numpy as np
import cv2

randVal = lambda : np.random.randint(0,high=255, dtype=np.uint8)
randomColor = lambda : (randVal(), randVal(), randVal())
targetColor = randomColor()

width = 500
height = 500
x = 20
y = 20
img = np.zeros((height, width,3), dtype=np.uint8)
target = np.full((100, 100,3), targetColor, dtype=np.uint8)
while y < height-100:
  x  = 20
  while x < width-100:
    img[y:y+100, x:x+100] = randomColor()
    x += 120
  y += 120

img[20:120, 20:120] = targetColor

This will create 2 images, img and target which are both random and in my test it gives something like this:

img:

target:

Now, I used the template matching as is, since in the documentation says that it may take 1 or 3 channels and it will do it correctly, but with the other method

res = cv2.matchTemplate(img[:,:,0], target[:,:,0], cv2.TM_SQDIFF_NORMED )
threshold = 0.00001
loc = np.where( res <= threshold )
img_rgb = img.copy()
for pt in zip(*loc[::-1]):
    cv2.rectangle(img_rgb, pt, (pt[0] + 100, pt[1] + 100), (0,0,255), 2)

This gives me the following result:

I hope this helps... I need to test the method you use in another version to see if it is a problem in my version or is something specific with this set of images...