如何应用基本矩阵将 3D 点从相机 1 的世界坐标转换为相机 2 的世界坐标

Question

Using OpenCV's cv2.stereoCalibrate I have calibrated a pair of cameras, one of them being a time-of-flight camera. So, I have the intrinsic calibration parameters and the essential / fundamental matrix.

Now I would like to project a point from the ToF camera to the 2D camera.

To convert image to world coordinates in the ToF camera, I did:

p = [(15, 15, 1)]
z = depth[p[0][0], p[0][1]]  # measured ToF depth for this single point

# from ToF image coordinates (including ToF depth!) to ToF world coordinates
invR_x_invM_x_uv1 = R_inv * cameraMatrix_inv_3 * p[0]
invR_x_tvec = R_inv * T
wcPoint = (z + invR_x_tvec[2]) / invR_x_invM_x_uv1[2] * invR_x_invM_x_uv1 - invR_x_tvec
wcPoint = wcPoint[:, -1]

So I have the point in world coordinates.

What I do not get, is (1) how to transform this point to the world coordinate system of the second camera, and then (2) how to project this point to image coordinates of the second camera. Can anybody point me to a OpenCV function in particular for (1)?

Answer 1

First, you should get into linear algebra. You can't multiply matrices elementwise. You have to use the dot product. The dot product of two matrices A and B is either written as A@B or A.dot(B) in Python.

If you have a stereo calibration and two fixed cameras, you also should have the extrinsic parameters of these cameras. The translation and rotation between these cameras and the world coordinate system (there is only one), which could be located at the position of one of the mentioned cameras, enables you to transform your 3D data to obtain the coordinates in the chosen coordinate system. I seriously suggest that you read the basics about camera projection in Hartley/Zisserman, Multiple View Geometry, freely available online.