OpenGL ES-从摄影机位置获取2D像素坐标

Question

I'm working with OpenGL ES (IOS) and OpenCV. I would like to know if there is a way to get 2D pixel coordinate from a GLKMatrix4 camera position ?

Indeed, I have a specific 3D object displayed with OpenGL which has only its size and its camera pose and I would like to convert it into the OpenCV reference Coordinate ie 2D pixel.

More precisely, I have an IOS app and I stream the camera color. This stream is displayed with OpenGL on my Ipad. On this display, I have a specific overlay which is a 3D object. At the same time, I do some some detection with OpenCV. Now, I would like to link the result of my detection thank to OpenCV in my frame (It's a rectangle draw) with the 3D object but for this one, I have only its camera position and its size.

I know the content of a 4x4 camera pose but I did not find any information about how I could convert it (if it is possible) to a 2D coordinate.

So briefly : Input : A 2D rectangle (get only 2D coordinate) and an 3D object (get only its camera position)

Is there a way ? Could you please help me ?

Thank

Answer 1

To get the onscreen projection of the vector you only need to multiply it by the matrix you are using in the openGL. The 3rd coordinate is simply discarded and is used only for the depth buffer (also interpolations but it really doesn't matter here).

In openGL the coordinate of the drawn pixel you will get by multiplying it with a matrix will be normalized to a coordinate system in range [-1,1] in all axises. That means a pixel that is on most top-left of the screen/view will have a coordinate (-1,1) , top-right (1,1) , center (0,0) ... Now corresponding to the buffer pixels this value must be denormalized:

You need to have the buffer width and height so then the pixel position is bufferX = ((x+1)/2.0) * width and bufferY = ((y+1)/2.0) * height . The results are floating values so you need to either round them to integers or typecast them. I have no idea which is done by the openGL though.

Answer 2

Since you're already using GLKit types and math, try the function GLKMathProject to convert from 3D model space to viewport (window/pixel 2D) space, and GLKMathUnproject if you also need the other way around. You'll need the (same) modelview and projection matrices that you're (presumably) passing to a shader for drawing your scene.