How to rectify a stereo image pair using opencv?

Question

I have set up two cameras and calibrated them separately, after that I calibrated both using stereoCalibrate function. Calibration seemed to work fine, as all returned reprojection errors are in the range of 0.4 to 0.5. Now i want to calculate a depth map out of the stereo image pair. As far as i could follow the tutorials, i first need to rectify my images, grayscale them and pass them to StereoBM (or any other matcher).

As far as i understand the code below should rectify and show two live images of the cameras.

... Load calibration matrices

//compute rectification
Mat R1, R2, P1, P2, Q;
stereoRectify(cameraMatrix_left, distortionCoefficients_left,
   cameraMatrix_right, distortionCoefficients_right,
   Size(left_camera.get(CAP_PROP_FRAME_WIDTH),
   left_camera.get(CAP_PROP_FRAME_HEIGHT)),
   R, T, R1, R2, P1, P2, Q, 0, 0.0,
   Size(left_camera.get(CAP_PROP_FRAME_WIDTH),
   left_camera.get(CAP_PROP_FRAME_HEIGHT)));

//compute undistortion
Mat rmap[2][2];
initUndistortRectifyMap(cameraMatrix_left, distortionCoefficients_left, R1, P1, Size(left_camera.get(CAP_PROP_FRAME_WIDTH), left_camera.get(CAP_PROP_FRAME_HEIGHT)), CV_16SC2, rmap[0][0], rmap[0][1]);
initUndistortRectifyMap(cameraMatrix_right, distortionCoefficients_right, R2, P2, Size(right_camera.get(CAP_PROP_FRAME_WIDTH), right_camera.get(CAP_PROP_FRAME_HEIGHT)), CV_16SC2, rmap[1][0], rmap[1][1]);

while (true) {
    if (!right_camera.read(capturedFrame_right))
        break;
    if (!left_camera.read(capturedFrame_left))
        break;

    remap(capturedFrame_left, drawFrame_left, rmap[0][0], rmap[0][1], INTER_LINEAR, BORDER_DEFAULT, Scalar());
    remap(capturedFrame_right, drawFrame_right, rmap[1][0], rmap[1][1], INTER_LINEAR, BORDER_DEFAULT, Scalar());

    cvtColor(drawFrame_left, grayScale_left, COLOR_RGB2GRAY);
    cvtColor(drawFrame_right, grayScale_right, COLOR_RGB2GRAY);

    imshow(RIGHT_CAMERA, grayScale_right);
    imshow(LEFT_CAMERA, grayScale_left);
}

I would expect that both images are rectified as shown on the documentation of stereoRectify .

However, they are simply not. There is a noticable vertical difference between both pictures. What did i miss?

Answer 1

You need to check your calibration is correct. Proper calibration matrix are loaded for that camera, it should provide a rectified image with the above code.

There were guidelines to do stereo calibration in the internet. Few of them listed here.

1. Specific distance of the calibration pattern from the camera shouldn't really matter. Calibration must be performed with fixed focus.
2. Having a checkerboard with more number of squares is beneficial as there are more corner points to extract. This enables us to get more 3D-2D point correspondences (Size of square shouldn't make a difference).
3. Different points of views and angles needed. Calibration detects focal length and distortion parameters by least square method, for which different angles of the checkerboard are needed to obtain a better solution.
4. Enough wide border around the calibration pattern is required.
5. A large pattern is better for cameras with larger baseline, as stereo overlap is lesser.
6. Some calibration toolboxes require the input of checkerboard dimensions to be either odd number x even number (eg 9x6) or vice versa (eg 8x7).
7. Preferably the checkerboard should cover at least half of the area of the image.
8. The calibration pattern must be moved to cover the entire FOV of the camera to obtain better calibration.Skew helps in focal length determination and moving the checkerboard towards the corners helps in determining the distortion co coefficients.
9. Good lighting conditions is of at most importance and is often overlooked.
10. Some sources say circles are easier to localize than corner points and using a circles pattern might lead to better calibration.
11. Humidity changes might affect normal paper that is used as it absorbs moisture. Thick paper must be used and the calibration pattern must be printed using a Laser printer and should be stuck on a glass backing preferably.