Mask Issue With Python OpenCV ORB Image Alignment

Question

I am trying to implement a Python (3.7) OpenCV (3.4.3) ORB image alignment. I normally do most of my processing with ImageMagick. But I need to do some image alignment and am trying to use Python OpenCV ORB. My script is based upon one from Satya Mallick's Learn OpenCV tutorial at https://www.learnopencv.com/image-alignment-feature-based-using-opencv-c-python/ .

However, I am trying to modify it to use a rigid alignment rather than a perspective homology and to filter the points using a mask to limit the difference in y values, since the images are nearly aligned already.

The mask approach was taken from a FLANN alignment code in the last example at https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html .

My script works fine, if I remove the matchesMask, which should provide the point filtering. (I have two other working scripts. One is similar, but just filters the points and ignores the mask. The other is based upon the ECC algorithm.)

However, I would like to understand why my code below is not working.

Perhaps the structure of my mask is incorrect in current versions of Python Opencv?

The error that I get is:

Traceback (most recent call last):
  File "warp_orb_rigid2_filter.py", line 92, in <module>
    imReg, m = alignImages(im, imReference)
  File "warp_orb_rigid2_filter.py", line 62, in alignImages
    imMatches = cv2.drawMatches(im1, keypoints1, im2, keypoints2, matches, None, **draw_params)
SystemError: <built-in function drawMatches> returned NULL without setting an error

Here is my code. The first arrow shows where the mask is created. The second arrow shows the line I have to remove to get the script to work. But then it ignores my filtering of points.

#!/bin/python3.7

import cv2
import numpy as np


MAX_FEATURES = 500
GOOD_MATCH_PERCENT = 0.15


def alignImages(im1, im2):

  # Convert images to grayscale
  im1Gray = cv2.cvtColor(im1, cv2.COLOR_BGR2GRAY)
  im2Gray = cv2.cvtColor(im2, cv2.COLOR_BGR2GRAY)

  # Detect ORB features and compute descriptors.
  orb = cv2.ORB_create(MAX_FEATURES)
  keypoints1, descriptors1 = orb.detectAndCompute(im1Gray, None)
  keypoints2, descriptors2 = orb.detectAndCompute(im2Gray, None)

  # Match features.
  matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING)
  matches = matcher.match(descriptors1, descriptors2, None)

  # Sort matches by score
  matches.sort(key=lambda x: x.distance, reverse=False)

  # Remove not so good matches
  numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
  matches = matches[:numGoodMatches]

  # Extract location of good matches and filter by diffy
  points1 = np.zeros((len(matches), 2), dtype=np.float32)
  points2 = np.zeros((len(matches), 2), dtype=np.float32)

  for i, match in enumerate(matches):
    points1[i, :] = keypoints1[match.queryIdx].pt
    points2[i, :] = keypoints2[match.trainIdx].pt

  # initialize empty arrays for newpoints1 and newpoints2 and mask
  newpoints1 = np.empty(shape=[0, 2])
  newpoints2 = np.empty(shape=[0, 2])
  matches_Mask = [0] * len(matches)

  # filter points by using mask    
  for i in range(len(matches)):
      pt1 = points1[i]
      pt2 = points2[i]
      pt1x, pt1y = zip(*[pt1])
      pt2x, pt2y = zip(*[pt2])
      diffy = np.float32( np.float32(pt2y) - np.float32(pt1y) )
      print(diffy)
      if abs(diffy) < 10.0:
        newpoints1 = np.append(newpoints1, [pt1], axis=0)
        newpoints2 = np.append(newpoints2, [pt2], axis=0)
        matches_Mask[i]=[1,0]  #<--- mask created
  print(matches_Mask)

  draw_params = dict(matchColor = (255,0,),
    singlePointColor = (255,255,0),
    matchesMask = matches_Mask, #<---- remove mask here
    flags = 0)

  # Draw top matches
  imMatches = cv2.drawMatches(im1, keypoints1, im2, keypoints2, matches, None, **draw_params)
  cv2.imwrite("/Users/fred/desktop/lena_matches.png", imMatches)


  # Find Affine Transformation
  # true means full affine, false means rigid (SRT)
  m = cv2.estimateRigidTransform(newpoints1,newpoints2,False)

  # Use affine transform to warp im1 to match im2
  height, width, channels = im2.shape
  im1Reg = cv2.warpAffine(im1, m, (width, height))

  return im1Reg, m


if __name__ == '__main__':

  # Read reference image
  refFilename = "/Users/fred/desktop/lena.png"
  print("Reading reference image : ", refFilename)
  imReference = cv2.imread(refFilename, cv2.IMREAD_COLOR)

  # Read image to be aligned
  imFilename = "/Users/fred/desktop/lena_r1.png"
  print("Reading image to align : ", imFilename);  
  im = cv2.imread(imFilename, cv2.IMREAD_COLOR)

  print("Aligning images ...")
  # Registered image will be stored in imReg. 
  # The estimated transform will be stored in m. 
  imReg, m = alignImages(im, imReference)

  # Write aligned image to disk. 
  outFilename = "/Users/fred/desktop/lena_r1_aligned.jpg"
  print("Saving aligned image : ", outFilename); 
  cv2.imwrite(outFilename, imReg)

  # Print estimated homography
  print("Estimated Affine Transform : \n",  m)

Here are my two images: lena and lena rotated by 1 degree. Note that these are not my actual images. These image have no diffy values > 10, but my actual images do.

I am trying to align and warp the rotated image to match the original lena image.

Answer 1

The way you are creating the mask is incorrect. It only needs to be a list with single numbers , with each number telling you whether you want to use that particular feature match.

Therefore, replace this line:

matches_Mask = [[0,0] for i in range(len(matches))]

With this:

matches_Mask = [0] * len(matches)

... so:

# matches_Mask = [[0,0] for i in range(len(matches))]
matches_Mask = [0] * len(matches)

This creates a list of 0s that is as long as the number of matches. Finally, you need to change writing to the mask with a single value:

  if abs(diffy) < 10.0:
    #matches_Mask[i]=[1,0]  #<--- mask created
    matches_Mask[i] = 1

I finally get this:

Estimated Affine Transform :
 [[ 1.00001187  0.01598318 -5.05963793]
  [-0.01598318  1.00001187 -0.86121051]]

Take note that the format of the mask is different depending on what matcher you use. In this case, you use brute force matching so the mask needs to be in the format that I just described.

If you used FLANN's knnMatch for example, then it will be a nested list of lists, with each element being a list that is k long. For example, if you had k=3 and five keypoints, it will be a list of five elements long, with each element being a three element list. Each element in the sub-list delineates what match you want to use for drawing.