OpenCV (python) fundamental and essential matrix do not agree

Question

I try to calibrate a stereo camera with OpenCV (python interface). I first calibrated the two cameras separately with calibrateCamera2 and then fed the parameters to stereoCalibrate

cv.StereoCalibrate(object_points, image_points_left, image_points_right, \
               point_counts, intrinsic_left, distortion_left,\
               intrinsic_right, distortion_right, \
               (IMGRES_X,IMGRES_Y), R, T, E, F, \
               term_crit=(cv.CV_TERMCRIT_ITER+cv.CV_TERMCRIT_EPS, 100, 1e-8),\
               flags=cv.CV_CALIB_FIX_INTRINSIC)

I check the result with the epipolar constraint (as described in the OpenCV book) and get an average error of around 0.0039.

In principle I should be able to relate the fundamental and the essential matrix with my camera matrices. So what I do is:

Mr = asarray(intrinsic_right,dtype=float64)
Ml = asarray(intrinsic_left,dtype=float64)
E = asarray(E)
F = asarray(F)
F2 = dot(dot(inv(Mr).T,E),inv(Ml))

However, the resulting matrix F2 does not at all resemble F. Is there something obvious that I am doing wrong? Help is much appreciated.

Edit: dot and inv are from numpy.

Answer 1

The E and F matrices returned by StereoCalibrate() are correct. F is defined up to scale, hence if you're going to compare the returned F and with the F matrix computed from E you need to normalize them to make sure both are at the same scale. So when you look at them they seem the same. StereoCalibrate() normalizes the returned F, thus you need to normalize the computed F2 as you have noted in one of your comments. I hope that makes it more clear why you need to do so.

Answer 2

I refer you to The Fundamental Matix Song ...

But seriously, perhaps it's something 'normalising" in the dot-products? The standard numpy dot function does seem to correctly act to split matrices into individual row and column vectors for multiplication.

For example, if I do:

A = mat(random.rand(3,3))
B = mat(random.rand(3,3))
dot(A,B) == A*B

Instead, I wonder if it helps to perform the straight-forward matrix multiplication as:

F2 = np.linalg.inv(Mr.T) * E * np.linalg.inv(Ml)

(NB I'm working with numpy matrices here)