How to convert back points in 2d to 3d with known orthogonal (camera) projection matrix?

Question

I have a numpy array with 2d points that I convert from 3d to 2d via the following equation:

https://wikimedia.org/api/rest_v1/media/math/render/svg/198f15da062c7ce00598d7a2f9bd8169d7042ed3

How can I convert the point back to 3D?

I used the top down view matrix that is in the image above. Found in Wikiperia: https://en.wikipedia.org/wiki/Orthographic_projection

#To 2D from 3d:
points2D = np.array([np.matmul(camera_pos, point) for point in points3D])[:,:2]

Answer 1

You cannot convert it back using just the projected points. Note that your projection basically is just looking at the (x,y) values and discarding the z value so there is no way to know what z was after doing this.

For instance, consider the points u = [1,2,3] and v=[1,2,-3] . These both project to [1,2,0] , so there is no way to know if we should make [1,2,0] into u or v when we try to invert (undo) the projection.

In terms of the matrix operation, this is because the projection matrices are not invertible (except the identity matrix).

You will need more information than just the projected points to be able to recover the original points.