OpenCV stereo vision 3D coordinates to 2D camera-plane projection different than triangulating 2D points to 3D

Question

I get an image point in the left camera ( pointL ) and the corresponding image point in the right camera ( pointR ) of my stereo camera using feature matching. The two cameras are parallel and are at the same "hight". There is only a x-translation between them.

I also know the projection matrices for each camera ( projL, projR ), which I got during calibration using initUndistortRectifyMap .

For triangulating the point, I call: triangulatePoints(projL, projR, pointL, pointR, pos3D) ( documentation ), where pos3D is the output 3D position of the object.

Now, I want to project the 3D-coordinates to the 2D-image of the left camera:

2Dpos = projL*3dPos

The resulting x-coordinate is correct. But the y-coodinate is about 20 pixels wrong.

How can I fix this?

Edit: Of course, I need to use homogeneous coordinates, in order to multiply it with the projection matrix (3x4). For that reason, I set:

3dPos[0] = x;
3dPos[1] = y;
3dPos[2] = z;
3dPos[3] = 1;

Is it wrong, to set 3dPos[3] to 1 ?

Note:

1. All images are remapped, I do this in a kind of preprocessing step.
2. Of course, I always use the homogeneous coordinates

Answer 1

You are likely projecting into the rectified camera. Need to apply the inverse of the rectification warp to obtain the point in the original (undistorted) linear camera coordinates, then apply distortion to get into the original image.