Fundamental Matrix for Point Correspondence

Question

I am estimating the fundamental matrix, by using

cv2.findFundamentalMat()

method of the OpenCV. I give keypoints which I get from flann matcher.

# Initiate ORB detector
orb = cv2.ORB_create()
# find the keypoints with ORB
keyPointsLeft = orb.detect(imgLeft,None)
keyPointsRight= orb.detect(imgRight, None)

# compute the descriptors with ORB
keyPointsLeft, descriptorsLeft = orb.compute(imgLeft, keyPointsLeft)
keyPointsRight, descriptorsRight = orb.compute(imgRight, keyPointsRight)
desLeft = np.float32(descriptorsLeft)
desRight = np.float32(descriptorsRight)
matches = flann.knnMatch(desLeft,desRight,k=2)
# Need to draw only good matches, so create a mask
matchesMask = [[0,0] for i in range(len(matches))]

# Apply ratio test
goodMatches = []
ptsLeft = []
ptsRight = []
for i,(m,n) in enumerate(matches):
    if m.distance < 0.7*n.distance:
        x1,y1 = keyPointsLeft[m.queryIdx].pt
        x2,y2 = keyPointsRight[n.trainIdx].pt
        matchesMask[i] = [1,0]
        goodMatches.append([m])
        ptsLeft.append(keyPointsLeft[m.queryIdx].pt)
        ptsRight.append(keyPointsRight[n.trainIdx].pt)

ptsLeft = np.int32(ptsLeft)
ptsRight = np.int32(ptsRight)
F, mask = cv2.findFundamentalMat(ptsLeft,ptsRight,cv2.FM_7POINT)

After finding the Fundamental matrix, I am trying to verify the point correspondences via epipolar constraint equation, which is :

(1) p1T * F * p2 = 0

So, lets say that we have point p1 (x1,y1) on left image and point p1 (x2,y2) on right image. So If I apply epipolar equation (1) to these points, I should get 0, or a closer number to 0.

So this equation, can be written like this :

(2) 0 = x1*x2*F[0][0] + x1*y2*F[0][1] + x1*F[0][2] + y1*x2*F[1][0] + y1*y2*F[1][1] + y1*F[1][2] + x2*F[2][0] + y2*F[2][1] +  F[2][2]

By using equation (2), I am trying to verify the points matched by flann with this equation. I printed the equation results and the points which are in the same row, since my input images have just translation between them.

Output :

p1(82,340) --> p2(74,340)
p1T * F * p2 =  -0.7662387780488729
p1(355,240) --> p2(354,240)
p1T * F * p2 =  -0.0047911844235635215
p1(354,229) --> p2(349,229)
p1T * F * p2 =  0.11662882831689814
p1(377,175) --> p2(372,175)
p1T * F * p2 =  0.3450325352994703
p1(319,227) --> p2(311,227)
p1T * F * p2 =  0.19119563752361657
p1(276,171) --> p2(273,171)
p1T * F * p2 =  0.251353775637849
p1(371,259) --> p2(366,259)
p1T * F * p2 =  -0.019570666111391688
......
....
...

With this output, I can't verify the points and my fundamental matrix. Equation (1) is not working in my case, sometimes it gives closer values to zero, sometimes not. Is there anything that I did wrong ?

Answer 1

I am not sure if equation 1 can be written as equation 2. I would definitely recommend using equation 1 and homogenous coordinates (z value is set to 1). You should consider normalizing the coordinates because X and Y would be much larger than Z. This can be done by computing a similarity transform established by Hartley, Zisserman and eg shown here: page 2 .

After that you could use the OpenCV Mat to directly compute the results. First, I would recommend that you try to use different methods for calculating the Fundamental Matrix. Since you are having a lot of feature points generated by ORB, I would use CV_FM_RANSAC eg with a reprojThreshold of 3 and confidence of 0.99. This could already do the trick.

A short code example in C++:

cv::Mat F = cv::findFundamentalMat(kp_left, kp_right, cv::RANSAC, 3.0, 0.999); # here you need to use your feature points cv::Point2f
point_left = cv::Mat(cv::Point3d(82,340,1));
point_right = cv::Mat(cv::Point3d(74,340,1));
cv::Mat distance = point_left.t()*F*point_right;

The distance is usually never exactly 0 but should now be close to it.