How to overlay/replace a part of an image with another using cv2?

Question

I have an image of a roof which covers a huge area! I am analyzing it for faults using segmentation. However the challenge is that sometimes the single shot of the roof doesn't have the details involved so its hard to analyse that. So we split it up into parts and take images of smaller parts and analyse that. Here is an example:

Entire roof shot

Part 1

Part 2

Part 3

So Now I want to analyse the parts for faults and replace it on the single shot. Here is how one of the analysed part looks:

Post Analysis

My thinking so far: If I can map the non analysed part to the roof in some way, I can use the same information and method to map the analysed part to the single shot too.

I have been able to do feature detection and identify the parts on the single shot however I am not sure as to how I should transform them to fit and replace the exact part on the single shot. Here is the feature detection:

Feature detection for image/template matching

I have been thinking of trying to stitch the smaller images to the larger one sequentially one after the other but I am concerned that might not work

PS: I am using Linux, Python3 and OpenCV 3.4.0

Also, the original single shot is not manually divided into the different parts. Since its a thermal cam, its not possible to have high res on the single shot so the cam takes different shots close up. Those parts are separate images and not cropped from the single shot.

EDIT: I am now trying an alternative approach where I detect faults in the part images and then crop those faults out to use as templates and then try feature detection using SIFT to match the fault as a template on the single shot. However, small faults are being matched to every thing other than itself :( Most of the larger faults are detected fine and medium ones dont do too good either. I think the reason is that the image is low res and the single shot pic doesnt replicate the faults on the closer shots and the number of features is also less on the single shot for template matching.

Answer 1

updated answer

My apologies! I misunderstood the question. So using your feature mapping method, you get a set of points on the original picture that correspond to a set of points in the new picture. Suppose the feature mapping is working perfectly, you could describe the correspondence as a mapping:

$$P_{new points} = T_{original to new } * P_{original} $$

where T_{original to new} is a transformation matrix . If you have this transformation matrix, then you can easily invert it to find the relationship between the "part" picture with the original one.

The challenge is that you currently have a complicated transformation matrix! The one you are looking for has a rotational component, translational component, scaling component, and most likely some skewing is happening too. I'm not sure the right approach for finding this matrix given the set of feature points.

If you don't mind doing some manual work, I recommend finding a set of lines that correspond on both pictures and use OpenCV's perspective transformation to find this matrix for you.

old, irrelevant answer

I have a KISS answer:

When you manually select "parts 1-3" from the "entire roof shot image", keep track of their initial size and location.

You don't need to compute the transform afterwards if you know it ahead of time. For starters, I recommend splitting the image into quartiles and work on overlaying the detections when you have this simple geometry. Once this works, approach the problem of arbitrarily sized boxes from the original picture.