WebGL, gl-matrix. How to get frustrum vertices from camera view and projection matrices?

Question

I am using the gl-matrix library. For linear algebra calculations

I have a view and projection matrix that I used for my camera in a 3d engine. The view is a lookat matrix.

const view =  mat4.lookAt(
    [],
    camera.eye,
    camera.target,
    camera.up
  );

const projection = mat4.perspective(
     [],
     Math.PI / 4,
     viewport.width / viewport.height,
     0.1,
     1000.0
    );

I want to get the 8 camera frustum vertices from this camera setup to use in a bounding box for a shadow map.

Answer 1

At perspective projection the projection matrix describes the mapping from 3D points in the world as they are seen from of a pinhole camera, to 2D points of the viewport.
The eye space coordinates in the camera frustum (a truncated pyramid) are mapped to a cube (the normalized device coordinates).
In normalized device space the corner points of the view volume are the corners of a cube with the left, bottom, near of (-1, -1, -1) and the right, top, far of (1, 1, 1).

To get the points in view space, the points in normalized device space have to be transformed by the inverse projection matrix, followed by a Perspective divide .
A point in view space can be transformed to a point in world space, by the inverse view matrix.

The inverse matrix can be computed by mat4.invert . The code to transform a point from normalized device space to world space may be as follows:

inv_view = mat4.invert([], view);
inv_proj = mat4.invert([], projection);

ndc_corner = vec4.set([], -1, -1, -1, 1); // (-1, -1, -1) left, bottom, near

view_corner_h = vec4.transformMat4([], ndc_corner, inv_proj);
view_corner = vec4.scale([], view_corner_h, 1/view_corner_h[3]);

world_corner = vec4.transformMat4([], view_corner, inv_view);