Converting Screen 2D to World 3D Coordinates

Question

I want to convert 2D screen coordinates to 3D world coordinates. I have searched a lot but I did not get any satisfying result.

Note: I am not using OpenGL nor any other graphics library.

Data which I have:

Screen X

Screen Y

Screen Height

Screen Width

Aspect Ratio

Answer 1

If you have the Camera world Matrix and Projection Matrix this is pretty simple.

If you don't have the world Matrix you can compute it from it's position and rotation.

worldMatrix = Translate(x, y, z) * RotateZ(z_angle) * RotateY(y_angle) * RotateX(x_angle);

Where translate returns the the 4x4 translation matrices and Rotate returns the 4x4 rotation matrices around the given axis.

The projection matrix can be calculated from the aspect ratio, field of view angle, and near and far planes.

This blog has a good explanation of how to calculate the projection matrix.

You can unproject the screen coordinates by doing:

mat = worldMatrix * inverse(ProjectionMatrix)
dir = transpose(mat) * <x_screen, y_screen, 0.5, 1>

dir /= mat[3] + mat[7] + mat[11] + mat[15]
dir -= camera.position

Your ray will point from the camera in the direction dir.

This should work, but it's not a super concreate example on how to do this.

Basically you just need to do the following steps:

calculate camera's worldMatrix
calculate camera's projection matrix
multiply worldMatrix with inverse projection matrix.

create a point <Screen_X_Value, Screen_Y_Value, SOME_POSITIVE_Z_VALUE, 1>
apply this "inverse" projection to your point.
then subtract the cameras position form this point.

The resulting vector is the direction from the camera. Any point along that ray are the 3D coordinates corresponding to your 2D screen coordinate.