使用 OpenCV SDK 从手机摄像头捕获图像以在 Unity 中进行全景拍摄

Question

How to capture images to create a panorama image using phone's camera(not unity camera) ? In other words, I'm trying to replicate the camera app on phone using Unity. I know a couple of methods on how to stitch images after they are captured but what I don't know is the "capture" process, i,e, how to determine if the phone is rotating left-to-right or right-to-left or not moving at all ? Should I be using the gyroscope readings ? If yes, what about the case when there is IMU drift ? Also, what is the approach used by most of the modern phones supporting panorama feature ?

Answer 1

Phones use sensor fusion and sometime also combine vision in order to determine how to do the stiching with regard to the motion. Most common is the use of Kalman filter where you have multiple sources of information in different measurements that can be aggregated to the desired variable. Basically what you are looking for is device orientation https://developer.android.com/reference/android/hardware/Sensor#TYPE_GEOMAGNETIC_ROTATION_VECTOR in Android for example. It uses Magnetometer, Accelerometer and Gyroscope 1. Magnetometer to estimate earth north 2. Accelerometer to estimate gravity direction 3. Gyroscope to compute the motion in dynamic time, the accelerometer and magnetometer will keep the gyroscope from drifting while the gyroscope helps to give valid results while the device in motion and the accelerometer evaluation of gravity direction is not accurate due to the linear acceleration. I can go on and on for the details but its not that important as its already implemented for you. This give you a rotation vector in reference to earth coordinate system. After you make the desired images or in real time you should perfect the results using some vision as the sensor accuracy will give you at least 2-3 degrees of error.

Hope it helps, Good luck with the project!