Computer Vision 3D transform from Images

Question

Here is a simplified (but gives the essence) of the problem. Suppose I have a box in space in some reference position/orientation and a calibrated camera C with known position/orientation. I take a picture of the box and can identify N feature points x_i on the projected image B.

Now suppose someone moves the box (rigid-body transform) a relatively small amount. I take a picture of the box and can again identify N feature points x*_i. I want to solve for the rigid-body transform T.

My strategy is to equivalently suppose the box did not move, and suppose I have another camera C* that is found by transforming camera C by the inverse of the transform T. So the N points x _i are the projected feature points on image B relative to camera C*.

So then I believe I can solve for the essential matrix E from the two sets of projected image points (provided I have enough--I think I need 8). (Since the cameras are calibrated I think I can just use essential matrix, not fundamental matrix?) From there I can use matrix decomposition to extract the rotation and translation transform that describes how the cameras differ. The inverse of that is the transform I want.

Does that sound like it will work? What happens if I can't find 8 feature points, but say only 3? Will I be able to get an estimate of the essential matrix or will it totally be wrong?

Answer 1

Yes, it will work and it is possible to solve even for some lost features as far as you are able to tell which is which. As far as I know you need at least 8 points like you said. What you described is how "Structure from motion" algorithms work. Please look it up on slide 3 in this lecture, and next at slide 20: The 8 point algorithm. It relates exactly to what you are talking about. If you realized all this without even knowing about structure form motion, then I am really impressed.

Here's the link to the lecture: https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws14-15/3DCV_lec06_SFM1.pdf