Warp perspective and stitch/overlap images (C++)

Question

I am detecting and matching features of a pair of images, using a typical detector-descriptor-matcher combination and then findHomography to produce a transformation matrix.

After this, I want the two images to be overlapped (the second one ( imgTrain ) over the first one ( imgQuery ), so I warp the second image using the transformation matrix using:

cv::Mat imgQuery, imgTrain;
...    
TRANSFORMATION_MATRIX = cv::findHomography(...)
...
cv::Mat imgTrainWarped;
cv::warpPerspective(imgTrain, imgTrainWarped, TRANSFORMATION_MATRIX, imgTrain.size());

From here on, I don't know how to produce an image that contains the original imgQuery with the warped imgTrainWarped on it. I consider two scenarios: 1) One where the size of the final image is the size of imgQuery 2) One where the size of the final image is big enough to accommodate both imgQuery and imgTrainWarped , since they overlap only partially, not completely. I understand this second case might have black/blank space somewhere around the images.

Answer 1

You should warp to a destination matrix that has the same dimensions as imgQuery after that, loop over the whole warped image and copy pixel to the first image, but only if the warped image actually holds a warped pixel. That is most easily done by warping an additional mask. Please try this:

cv::Mat imgMask = cv::Mat(imgTrain.size(), CV_8UC1, cv::Scalar(255));
cv::Mat imgMaskWarped;
cv::warpPerspective(imgMask , imgMaskWarped, TRANSFORMATION_MATRIX, imgQuery.size());

cv::Mat imgTrainWarped;
cv::warpPerspective(imgTrain, imgTrainWarped, TRANSFORMATION_MATRIX, imgQuery.size());

// now copy only masked pixel:
imgTrainWarped.copyTo(imgQuery, imgMaskWarped);

please try and tell whether this is ok and solves scenario 1 . For scenario 2 you would test how big the image must be before warping (by using the transformation) and copy both images to a destination image big enough.

Answer 2

Are you trying to create a panoramic image out of two overlapping pictures taken from the same viewpoint in different directions? If so, I am concerned about the "the second one over the first one" part. The correct way to stitch the panorama together is to cut both images off down the central line (symmetry axis) of the overlapping part, and not to add a part of one image to the (whole) other one.

Answer 3

Accepted answer works but could be done easier with using BORDER_TRANSPARENT :

cv::warpPerspective(imgTrain, imgQuery, TRANSFORMATION_MATRIX, imgQuery.size(), INTER_LINEAR, BORDER_TRANSPARENT);

When using BORDER_TRANSPARENT the source pixel of imgQuery remains untouched.

Answer 4

对于 OpenCV 4 INTER_LINEAR 和 BORDER_TRANSPARENT 可以通过使用 cv::InterpolationFlags::INTER_LINEAR, cv::BorderTypes::BORDER_TRANSPARENT 解决，例如

cv::warpPerspective(imgTrain, imgQuery, TRANSFORMATION_MATRIX, imgQuery.size(), cv::InterpolationFlags::INTER_LINEAR, cv::BorderTypes::BORDER_TRANSPARENT);