Reprojection of calibrateCamera and projectPoints

Question

I am using OpenCV's calibrateCamera and am trying to understand how it calculates the reprojection error as well as what this error represents. It appears to be the RMS of the (Euclidean) distance between the projected points and measured image points--is this right? However, what does it mean for the final reprojection error to be "minimized"? Does calibrateCamera() explicitly use the function projectPoints() to find the projected points?

Answer 1

The reprojection error is the error (Euclidean distance for example) between the 3D points reprojected with the estimated intrinsic and extrinsic matrices and the 2D image points detected by some image processing techniques (corners of the chessboard pattern for example).

The final reprojection error is minimized because the problem to estimate the set of intrinsic and extrinsic parameters is a non-linear problem and thus you have to find the set of parameters that minimizes this reprojection error iteratively.

More information: A Flexible New Technique for Camera Calibration ; Zhengyou Zhang ; 2000 .

Answer 2

I am referring to OpenCV version 3.1.0, here you find doc for calibrateCamera http://docs.opencv.org/3.1.0/d9/d0c/group__calib3d.html#ga687a1ab946686f0d85ae0363b5af1d7b and the doc says (bold is mine):

The algorithm performs the following steps:

Compute the initial intrinsic parameters (the option only available for planar calibration patterns) or read them from the input parameters. The distortion coefficients are all set to zeros initially unless some of CV_CALIB_FIX_K? are specified.

Estimate the initial camera pose as if the intrinsic parameters have been already known. This is done using solvePnP .

Run the global Levenberg-Marquardt optimization algorithm to minimize the reprojection error, that is, the total sum of squared distances between the observed feature points imagePoints and the projected (using the current estimates for camera parameters and the poses) object points objectPoints. See projectPoints for details.

The function returns the final re-projection error .

Anyway, instead of relying on the doc, I will prefer to have a look at the code:

https://github.com/Itseez/opencv/blob/3.1.0/modules/calib3d/src/calibration.cpp#L3298

Answer 3

When you use projectPoints , You need to calculate RMS by hand after the reprojection. This might help. OPENCV: Calibratecamera 2 reprojection error and custom computed one not agree

Answer 4

The reprojection error is not defined 100% mathematically in literature. Formally, a single reprojection error is the 2d vector relating measured to projected pixel coordinates.

OpenCV and most other software and related bundle adjustment algorithms use the sum of squares of Euclidean lengths of these 2d vectors as the objective function during optimization.

As Geoff and Alessandro Jacopson have pointed out, the return value of cv::calibrateCamera() is the RMS of Euclidean errors (contrary to the doc of v3.1). This quantity is directly related to the objective function value, but not exactly the same.

Other definitions of the reprojection error include the mean Euclidean length and the median of Euclidean lengths. Both are legitimate and care must be given when comparing values.

An in-depth article on this topic can be found here: https://calib.io/blogs/knowledge-base/understanding-reprojection-errors

Answer 5

Here is the reprojection error calculation from the OpenCV code calibrate.cpp line 1629:

return std::sqrt(reprojErr/total);

Where total is the sum of all points for all images.