Python, OpenCV to find and overlay/replace portions of an image

Question

Need to find and replace some portions in an image, with other images.

The original image is shown below. I want to find and replace:

blue squares to blue triangles
orange squares to orange octagons

the image files are in place as:

original.jpg, blue square.jpg, orange square.jpg, blue triangle.jpg, orange octagon.jpg

using cv2, I can find the locations of the target images.

import cv2
import numpy as np

img_original = cv2.imread("C:\\original.jpg")

img_to_replace_0 = cv2.imread("C:\\blue square.jpg")
img_to_replace_1 = cv2.imread("C:\\orange square.jpg")

img_overlay_0 = cv2.imread("C:\\blue triangle.jpg")
img_overlay_1 = cv2.imread("C:\\orange octagon.jpg")

res_0 = cv2.matchTemplate(img_original, img_to_replace_0, cv2.TM_CCOEFF_NORMED)
res_1 = cv2.matchTemplate(img_original, img_to_replace_1, cv2.TM_CCOEFF_NORMED)

threshold = 0.80
loc_0 = np.where (res_0 >= threshold)
loc_1 = np.where (res_1 >= threshold)

bl_0 = list(loc_0)
bl_1 = list(loc_1)

print bl_0
print bl_1

the output is:

[array([106, 294, 477]), array([17, 18, 21])]
[array([ 22, 210, 393]), array([16, 17, 20])]

What's the best way to proceed further? Is CV2 the best tool for this case? Thank you.

Answer 1

Yes, it is possible to continue from that point. You will get the locations of each of then, first array is y axis and the second array the x axis. This position is the top left corner of the object.

What could be aproblem here is that the templates have different sizes.. so it may overlap with other parts of the image which is not happening in the original image and I do not know if this is a problem. So you may have to make sure all of them are equally big or that they have enough space if this is a problem. This could be solve, scaling the images to be of the same size for example.

Anyways, you may achieve the replacement like this:

import cv2
import numpy as np

# load the data
img = cv2.imread('scene.jpg')
blueTri = cv2.imread('blueTri.jpg')
blueSq = cv2.imread('blueSq.jpg')
bgColor = (255,255,255)

# find the matching rectangles
res = cv2.matchTemplate(img, blueSq, cv2.TM_CCOEFF_NORMED)
threshold = 0.98 # I used a higher threshold, because it was giving several "good" locations that are next to each other
loc = np.where (res >= threshold)

# process the positions
for i in range(len(loc[0])):
  # it is given as (y,x)
  pos = (loc[0][i], loc[1][i])
  # delete the blue squares
  img[pos[0]:pos[0]+blueSq.shape[0] , pos[1]:pos[1]+blueSq.shape[1] ] = bgColor
  # put the new blue triangle
  img[pos[0]:pos[0]+blueTri.shape[0] , pos[1]:pos[1]+blueTri.shape[1] ] = blueTri


cv2.imshow("Frame", img)
cv2.waitKey(0)
cv2.destroyAllWindows()

And here is the result of blue squares to blue triangles:

And for both changes as in your example:

If the sizes of the two templates are equal (ie the image you look for and the one it will be replace with), you can reduce one step and change the for loop to:

# process the positions
for i in range(len(loc[0])):
  # it is given as (y,x)
  pos = (loc[0][i], loc[1][i])
  # put the new blue triangle
  img[pos[0]:pos[0]+blueTri.shape[0] , pos[1]:pos[1]+blueTri.shape[1] ] = blueTri

---

UPDATE

To move the shape you are copying you can shift the whole ROI, anyways you already have deleted the old one so it will be ok. Just don't forget to check the limitsof the image. For that you can do the following:

for i in range(len(loc[0])):
  posToDelete = (loc[0][i], loc[1][i])
  posToAdd = (loc[0][i] -10, loc[1][i]+15) # 10 pixels up and 15 to the right
  posToAdd = (max(0, min(posToAdd[0],img.shape[0]-1 -blueTri.shape[0] )) , max(0, min(posToAdd[1],img.shape[1]-1-blueTri.shape[1]))) # sanity check to make sure it is inside the borders
  img[posToDelete[0]:posToDelete[0]+blueSq.shape[0] , posToDelete[1]:posToDelete[1]+blueSq.shape[1] ] = bgColor
  img[posToAdd[0]:posToAdd[0]+blueTri.shape[0] , posToAdd[1]:posToAdd[1]+blueTri.shape[1] ] = blueTri