为什么epiplolar geometry/sfm在python opencv中没有给出正确的值

Question

I tried to find rotation/translation between two images.

For simplest case, I use two exact same images, and checked whether it gives 0 translations and rotations(identity matrix).

However it does not give the results that I expected. why???

ORB feature is used and ten matched features are used to find Essential matrix and R/t. Result is (they are identical images):

t =  [[ 0.57735027] [-0.57735027] [ 0.57735027]]
r = [[-0.33333333 -0.66666667  0.66666667]
     [-0.66666667 -0.33333333 -0.66666667]
     [ 0.66666667 -0.66666667 -0.33333333]]

What I expected is :

t = [[0, 0, 0]]
r = [[1, 0, 0], [0, 1, 0], [0, 0, 0]]

Why it does not give strange results?

    orb = cv2.ORB_create()
    img1 = self.img1
    img2 = self.img2

    gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
    gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)

    kpts1, descs1 = orb.detectAndCompute(gray1, None)
    kpts2, descs2 = orb.detectAndCompute(gray2, None)

    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
    matches = bf.match(descs1, descs2)
    dmatches = sorted(matches, key=lambda x: x.distance)

    src_pts = np.float32([kpts1[m.queryIdx].pt for m in dmatches]).reshape(-1, 1, 2)
    src_pts = src_pts[0:10]
    dst_pts = np.float32([kpts2[m.trainIdx].pt for m in dmatches]).reshape(-1, 1, 2)
    dst_pts = dst_pts[0:10]

    K = np.array([[842.102288, 0., 263.697271],
                  [0., 833.300569, 536.024168],
                  [0., 0., 1.]])

    E, mask2 = cv2.findEssentialMat(src_pts, dst_pts, K, cv2.RANSAC, 0.999, 1.0);
    points, R, t, mask = cv2.recoverPose(E, src_pts, dst_pts)

test image (image1=image2=img)

Answer 1

sometimes we need to test to find the problem, so it will be very useful if you put the link to the image.

However, there are a few things you should know:

• By default on OpenCV, the maximum number of keypoints to be detected using the ORB detector is equal to 500 keypoints
• By calculating the essential matrix , you get inliers as well as outliers which are represented by a mask, but according to your source code, you don't take it into consideration, which affects the result of recoverpose
• Recoverpose also needs the camera matrix

So for a first step, I ask you to test this code, while waiting for the publication of your image

    orb = cv2.ORB_create(nfeatures = 2000)
    img1 = self.img1
    img2 = self.img2

    gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
    gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)

    kpts1, descs1 = orb.detectAndCompute(gray1, None)
    kpts2, descs2 = orb.detectAndCompute(gray2, None)

    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
    matches = bf.match(descs1, descs2)
    dmatches = sorted(matches, key=lambda x: x.distance)

    src_pts = np.float32([kpts1[m.queryIdx].pt for m in dmatches]).reshape(-1, 1, 2)
    dst_pts = np.float32([kpts2[m.trainIdx].pt for m in dmatches]).reshape(-1, 1, 2)

    K = np.array([[842.102288, 0., 263.697271],
                  [0., 833.300569, 536.024168],
                  [0., 0., 1.]])

    E, inliers_mask_E = cv2.findEssentialMat(src_pts, dst_pts, K method = cv2.RANSAC, prob = 0.999, threshold = 1.0);
    points, R, t, inliers_mask_RP = cv2.recoverPose(E, src_pts, dst_pts, cameraMatrix = K, mask = inliers_mask_E)