How to improve template matching result?

Question

I'd like to detect juice boxes of a certain type on a store shelf.

Example of image of shelf:

Example of image of box:

My code approach so far:

import numpy as np
import cv2
import json
from skimage.measure import label
import sys
import matplotlib.pyplot as plt
from PIL import Image

def return_boxes(img, template):
    w, h = template.shape[::-1]
    res_img = np.int16(img.copy())

    img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    match = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
    
    loc = np.where(match >= 0.4)
    lbl, n = label(match >= 0.4, connectivity=2, return_num=True)
    
    lbl = np.int16([np.round(np.mean(np.argwhere(lbl == i), axis=0)) for i in range(1, n + 1)])
    
    centers = [[pt[0]+w//2, pt[1]+h//2] for pt in lbl]
    
    for pt in zip(*loc[::-1]):
        cv2.rectangle(res_img, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
    
    return res_img, centers
    
    
    
def color(img):
    return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)

def gray(img):
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)    
    
    
def predict_image(img, query):
    template = gray(query)
    res_img, centers = return_boxes(img, template)
    return res_img, centers

And the result

is far from accurate. How can I improve it? I can obviously change threshold or resize the image/template, but it does not seem universal and robust. Are there ways to improve such matching for different shelves and boxes? Some preprocessing of image perhaps.

Answer 1

as far as i know, template matching could only get same image with same width and length and

Answer 2

cv2 can't correctly find images. Sometimes cv2 find template in empty place. You cant check or controll it by cv2 methods. I think u must check results using pillow by pixels cpmpare.