Stereo Calibration in Open CV and 3D Coordinates

Question

I am a beginner in OpenCV and am currently working on a project requiring the exact mapping of pixels to centimetres/millimetres/any real world unit.

I have performed Stereo Calibration in OpenCV followed by Stereo Rectification. And thus I have obtained both the intrinsic and extrinsic parameters.

While performing Stereo Calibration, I have no where entered the exact square size (25mm by 25mm) of the chessboard patter. I have only entered the number of horizontal and vertical inner corners as well as number of boards. So will this have an effect when I reproject to 3D? If so, how do I include the square size in the StereoCalibrate function.

Secondly, the values fx and fy obtained from the camera matrix are both 513.86 while from the EXIF data it comes out to be 3.7mm. So, what is the exact relation between both?

Thirdly, I have used reprojectImageTo3D and obtained the 3d world coordinates. Now what are the exact units of these coordinates (cm/mm/inches/etc.)?

Basically, I want to obtain the exact mapping from pixel to real world units and after lots of reading and searching, I have not been able to do so. Kindly help me with this issue.

Answer 1

I agree with the other answers for the first two questions. But just to clarify further and elaborate more,

1)

Look at the parameter 'squareSize' in the code block below.

This is where you give the size of the square. What this does is, it calculates the position of each corner in the chess board and pushes it into a vector of Points, called objectPoints. This is possible because a chessboard has a regular pattern.

The vector objectPoints then becomes the first parameter of your StereoCalibrate function.

Also, please remember that whatever units you specify for squareSize, your calibration results (except fx and fy of course), will be in the same units.

    for( j = 0; j < boardSize.height; j++ ){
        for( k = 0; k < boardSize.width; k++ ){
            objectPoints.push_back(Point3f(j*squareSize, k*squareSize, 0));
        }
    }

2)

fx and fy that you get, will be in pixels. In order to get the focal length in mm, you need to multiply it with a scaling factor.

The relationship between fx, fy and your actual focal length in mm is given by the size of your camera sensor. You should be able to find this parameter in the camera specs.

The scaling factor that I mentioned above will be the number of pixels per mm for your camera sensor.

3)

The 3D world coordinates that you get, are exactly what you mentioned. Coordinates . Referenced from some point in your system. They are the x , y and z coordinates of the object points. Coordinates don't have units!

Answer 2

I can answer two first questions so far:

1: First param in cv::stereoCalibrate is vector of object points, you can fill it by using following function:

void CalcBoardCornerPositions(cv::Size boardSize, double squareSize, std::vector<cv::Point3f>& corners) {
    corners.clear();
    for( int i = 0; i < boardSize.height; ++i )
        for( int j = 0; j < boardSize.width; ++j )
            corners.push_back(cv::Point3f(float( j*squareSize ), float( i*squareSize ), 0)); 
}

And then use it:

 std::vector<std::vector<cv::Point3f> > objectPoints(1);
 CalcBoardCornerPositions(boardSize, squareSize, objectPoints[0]);
 cv::stereoCalibrate(objectPoints, imagePointsA, imagePointsB, _cameraMatrixA, /// etc

2: The camera matrix fx and fy are in fact focal length multiplied by density of pixels in direction x and y respectively. That means

fx = f * cx

fy = f * cy

where cx = imageSize.width / sensorSize.width and cy = imageSize.height / sensorSize.height. Usually cx == cy in modern cameras. Size of your camera sensor you can find in camera manual.

Answer 3

1. The first parameter of function StereoCalibrate is the actual dimension of your calibration pattern which you need to create by yourself. For example if you have a chessboard grid 6x5 with squares of 3 mm you need to create a matrix that contain the position of each corners expressed in a reference system on the chessboard ex [(0,0,0)(0,3,0)(0,6,0)...] and so on.
2. The focal length is expressed in pixel dimension. You should just check the actual dimension of the pixel (typically in the order of micrometer) and multiply fx and fy by it.
3. The units depends on how you created the calibration pattern (question number one). In the example I made the chessboard has squares of 3 mm so the position of the corners in the grid are expressed in millimeters. That is the units of your results.