OpenGL - Frustum not culling polygons beyond far plane

Question

I have implemented frustum culling and am checking the bounding box for its intersection with the frustum planes. I added the ability to pause frustum updates which lets me see if the frustum culling has been working correctly. When I turn around after I have paused it, nothing renders behind me and to the left and right side, they taper off as well just as you would expect. Beyond the clip distance (far plane), they still render and I am not sure whether it is a problem with my frustum updating or bounding box checking code or I am using the wrong matrix or what. As I put the distance in the projection matrix at 3000.0f, it still says that bounding boxes well past that are still in the frustum, which isn't the case.

Here is where I create my modelview matrix:

projectionMatrix = glm::perspective(newFOV, 4.0f / 3.0f, 0.1f, 3000.0f);

viewMatrix = glm::mat4(1.0);
viewMatrix = glm::scale(viewMatrix, glm::vec3(1.0, 1.0, -1.0));
viewMatrix = glm::rotate(viewMatrix, anglePitch, glm::vec3(1.0, 0.0, 0.0));
viewMatrix = glm::rotate(viewMatrix, angleYaw, glm::vec3(0.0, 1.0, 0.0));
viewMatrix = glm::translate(viewMatrix, glm::vec3(-x, -y, -z));

modelViewProjectiomMatrix = projectionMatrix * viewMatrix;

The reason I scale it by -1 in the Z direction is because the levels were designed to be rendered with DirectX so I reverse the Z direction.

Here is where I update my frustum:

void CFrustum::calculateFrustum()
{
    glm::mat4 mat = camera.getModelViewProjectionMatrix();

    // Calculate the LEFT side
    m_Frustum[LEFT][A] = (mat[0][3]) + (mat[0][0]);
    m_Frustum[LEFT][B] = (mat[1][3]) + (mat[1][0]);
    m_Frustum[LEFT][C] = (mat[2][3]) + (mat[2][0]);
    m_Frustum[LEFT][D] = (mat[3][3]) + (mat[3][0]);

    // Calculate the RIGHT side
    m_Frustum[RIGHT][A] = (mat[0][3]) - (mat[0][0]);
    m_Frustum[RIGHT][B] = (mat[1][3]) - (mat[1][0]);
    m_Frustum[RIGHT][C] = (mat[2][3]) - (mat[2][0]);
    m_Frustum[RIGHT][D] = (mat[3][3]) - (mat[3][0]);

    // Calculate the TOP side
    m_Frustum[TOP][A] = (mat[0][3]) - (mat[0][1]);
    m_Frustum[TOP][B] = (mat[1][3]) - (mat[1][1]);
    m_Frustum[TOP][C] = (mat[2][3]) - (mat[2][1]);
    m_Frustum[TOP][D] = (mat[3][3]) - (mat[3][1]);

    // Calculate the BOTTOM side
    m_Frustum[BOTTOM][A] = (mat[0][3]) + (mat[0][1]);
    m_Frustum[BOTTOM][B] = (mat[1][3]) + (mat[1][1]);
    m_Frustum[BOTTOM][C] = (mat[2][3]) + (mat[2][1]);
    m_Frustum[BOTTOM][D] = (mat[3][3]) + (mat[3][1]);

    // Calculate the FRONT side
    m_Frustum[FRONT][A] = (mat[0][3]) + (mat[0][2]);
    m_Frustum[FRONT][B] = (mat[1][3]) + (mat[1][2]);
    m_Frustum[FRONT][C] = (mat[2][3]) + (mat[2][2]);
    m_Frustum[FRONT][D] = (mat[3][3]) + (mat[3][2]);

    // Calculate the BACK side
    m_Frustum[BACK][A] = (mat[0][3]) - (mat[0][2]);
    m_Frustum[BACK][B] = (mat[1][3]) - (mat[1][2]);
    m_Frustum[BACK][C] = (mat[2][3]) - (mat[2][2]);
    m_Frustum[BACK][D] = (mat[3][3]) - (mat[3][2]);

    // Normalize all the sides
    NormalizePlane(m_Frustum, LEFT);
    NormalizePlane(m_Frustum, RIGHT);
    NormalizePlane(m_Frustum, TOP);
    NormalizePlane(m_Frustum, BOTTOM);
    NormalizePlane(m_Frustum, FRONT);
    NormalizePlane(m_Frustum, BACK);
}

And finally, where I check the bounding box:

bool CFrustum::BoxInFrustum( float x, float y, float z, float x2, float y2, float z2)
{
    // Go through all of the corners of the box and check then again each plane
    // in the frustum.  If all of them are behind one of the planes, then it most
    // like is not in the frustum.
    for(int i = 0; i < 6; i++ )
    {
        if(m_Frustum[i][A] * x  + m_Frustum[i][B] * y  + m_Frustum[i][C] * z  + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y  + m_Frustum[i][C] * z  + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x  + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z  + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z  + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x  + m_Frustum[i][B] * y  + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y  + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x  + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0)  continue;
        if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0)  continue;

        // If we get here, it isn't in the frustum
        return false;
    }

    // Return a true for the box being inside of the frustum
    return true;
}

Answer 1

I've noticed a few things, particularly with how you set up the projection matrix. For starters, gluProject doesn't return a value, unless you're using some kind of wrapper or weird api. gluLookAt is used more often.

Next, assuming the scale, rotate, and translate functions are intended to change the modelview matrix, you need to reverse their order. OpenGL doesn't actually move objects around; instead it effectively moves the origin around, and renders each object using the new definition of <0,0,0>. Thus you 'move' to where you want it to render, then you rotate the axes as needed, then you stretch out the grid.

As for the clipping problem, you may want to give glClipPlane() a good look over. If everything else mostly works, but there seems to be some rounding error, try changing the near clipping plane in your perspective(,,,) function from 0.1 to 1.0 (smaller values tend to mess with the z-buffer).

I see a lot of unfamiliar syntax, so I think you're using some kind of wrapper; but here are some (Qt) code fragments from my own GL project that I use. Might help, dunno:

//This gets called during resize, as well as once during initialization
void GLWidget::resizeGL(int width, int height) {
  int side = qMin(width, height);
  padX = (width-side)/2.0;
  padY = (height-side)/2.0;
  glViewport(padX, padY, side, side);

  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  gluPerspective(60.0, 1.0, 1.0, 2400.0);

  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
}

//This fragment gets called at the top of every paint event:
  glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

  glPushMatrix();

  glLightfv(GL_LIGHT0, GL_POSITION, FV0001);

  camMain.stepVars();

  gluLookAt(camMain.Pos[0],camMain.Pos[1],camMain.Pos[2],
            camMain.Aim[0],camMain.Aim[1],camMain.Aim[2],   
            0.0,1.0,0.0);

  glPolygonMode(GL_FRONT_AND_BACK, drawMode);

//And this fragment represents a typical draw event
void GLWidget::drawFleet(tFleet* tIn) {
  if (tIn->firstShip != 0){
    glPushMatrix();

    glTranslatef(tIn->Pos[0], tIn->Pos[1], tIn->Pos[2]);
    glRotatef(tIn->Yaw, 0.0, 1.0, 0.0);
    glRotatef(tIn->Pitch,0,0,1);

    drawShip(tIn->firstShip);

    glPopMatrix();
  }
}

I'm working on the assumption that you're newish to GL, so my apologies if I come off as a little pedantic.

Answer 2

I had the same problem.

Given Vinny Rose's answer, I checked the function that creates a normalized plane, and found an error.

This is the corrected version, with the incorrect calculation commented out:

plane plane_normalized(float A, float B, float C, float D) {
    // Wrong, this is not a 4D vector
    // float nf = 1.0f / sqrtf(A * A + B * B + C * C + D * D);

    // Correct
    float nf = 1.0f / sqrtf(A * A + B * B + C * C);

    return (plane) {{
        nf * A,
        nf * B,
        nf * C,
        nf * D
    }};
}

My guess is that your NormalizePlane function does something similar.

The point of normalizing is to have a plane in Hessian normal form so that we can do easy half-space tests. If you normalize the plane as you would a four-dimensional vector, the normal direction [A, B, C] is still correct but the offset D is not.

I think you'd get correct results when testing points against the top, bottom, left and right planes because they pass through the origin, and the near plane might be close enough to not notice. (Bounding sphere tests would fail.)

The frustum cull worked as expected for me when I restored the correct normalization.

Answer 3

Here's what I think is happening: The far plane is getting defined correctly but in my testing the D value of that plane is coming out much too small. So objects are getting accepted as being on the correct side of the far plane because the math is forcing the far plane to actually be much farther away than you want.

Try a different approach: (http://www.lighthouse3d.com/tutorials/view-frustum-culling/geometric-approach-extracting-the-planes/)

float tang = tanf(fov * PI / 360.0f);
float nh = near * tang; // near height
float nw = nh * aspect; // near width
float fh = far * tang; // far height
float fw = fh * aspect; // far width

glm::vec3 p,nc,fc,X,Y,Z,Xnw,Ynh;

//camera position
p = glm::vec3(viewMatrix[3][0],viewMatrix[3][1],viewMatrix[3][2]);

// the left vector
glm::vec3 X = glm::vec3(viewMatrix[0][0], viewMatrix[1][0], viewMatrix[2][0]);
// the up vector
glm::vec3 Y = glm::vec3(viewMatrix[0][1], viewMatrix[1][1], viewMatrix[2][1]);
// the look vector
glm::vec3 Z = glm::vec3(viewMatrix[0][2], viewMatrix[1][2], viewMatrix[2][2]);

nc = p - Z * near; // center of the near plane
fc = p - Z * far; // center of the far plane

// the distance to get to the left or right edge of the near plane from nc
Xnw = X * nw;
// the distance to get to top or bottom of the near plane from nc
Ynh = Y * nh;
// the distance to get to the left or right edge of the far plane from fc
Xfw = X * fw;
// the distance to get to top or bottom of the far plane from fc
Yfh = Y * fh;

ntl = nc + Ynh - Xnw; // "near top left"
ntr = nc + Ynh + Xnw; // "near top right" and so on
nbl = nc - Ynh - Xnw;
nbr = nc - Ynh + Xnw;

ftl = fc + Yfh - Xfw;
ftr = fc + Yfh + Xfw;
fbl = fc - Yfh - Xfw;
fbr = fc - Yfh + Xfw;

m_Frustum[TOP] = planeWithPoints(ntr,ntl,ftl);
m_Frustum[BOTTOM] = planeWithPoints(nbl,nbr,fbr);
m_Frustum[LEFT] = planeWithPoints(ntl,nbl,fbl);
m_Frustum[RIGHT] = planeWithPoints(nbr,ntr,fbr);
m_Frustum[FRONT] = planeWithPoints(ntl,ntr,nbr);
m_Frustum[BACK] = planeWithPoints(ftr,ftl,fbl);

// Normalize all the sides
NormalizePlane(m_Frustum, LEFT);
NormalizePlane(m_Frustum, RIGHT);
NormalizePlane(m_Frustum, TOP);
NormalizePlane(m_Frustum, BOTTOM);
NormalizePlane(m_Frustum, FRONT);
NormalizePlane(m_Frustum, BACK);

Then planeWithPoints would be something like:

planeWithPoints(glm::vec3 a, glm::vec3 b, glm::vec3 c){
    double A = a.y * (b.z - c.z) + b.y * (c.z - a.z) + c.y * (a.z - b.z);
    double B = a.z * (b.x - c.x) + b.z * (c.x - a.x) + c.z * (a.x - b.x);
    double C = a.x * (b.y - c.y) + b.x * (c.y - a.y) + c.x * (a.y - b.y);
    double D = -(a.x * (b.y * c.z - c.y * b.z) + b.x * (c.y * a.z - a.y * c.z) + c.x * (a.y * b.z - b.y * a.z));
    return glm::vec4(A,B,C,D);
}

I didn't test any of the above. But the original reference is there if you need it.

Previous Answer: OpenGL and GLSL matrices are stored and accessed in column-major order when the matrix is represented by a 2D array. This is also true with GLM as they follow the GLSL standards.

You need to change your frustum creation to the following.

 // Calculate the LEFT side (column1 + column4) m_Frustum[LEFT][A] = (mat[3][0]) + (mat[0][0]); m_Frustum[LEFT][B] = (mat[3][1]) + (mat[0][1]); m_Frustum[LEFT][C] = (mat[3][2]) + (mat[0][2]); m_Frustum[LEFT][D] = (mat[3][3]) + (mat[0][3]); // Calculate the RIGHT side (-column1 + column4) m_Frustum[RIGHT][A] = (mat[3][0]) - (mat[0][0]); m_Frustum[RIGHT][B] = (mat[3][1]) - (mat[0][1]); m_Frustum[RIGHT][C] = (mat[3][2]) - (mat[0][2]); m_Frustum[RIGHT][D] = (mat[3][3]) - (mat[0][3]); // Calculate the TOP side (-column2 + column4) m_Frustum[TOP][A] = (mat[3][0]) - (mat[1][0]); m_Frustum[TOP][B] = (mat[3][1]) - (mat[1][1]); m_Frustum[TOP][C] = (mat[3][2]) - (mat[1][2]); m_Frustum[TOP][D] = (mat[3][3]) - (mat[1][3]); // Calculate the BOTTOM side (column2 + column4) m_Frustum[BOTTOM][A] = (mat[3][0]) + (mat[1][0]); m_Frustum[BOTTOM][B] = (mat[3][1]) + (mat[1][1]); m_Frustum[BOTTOM][C] = (mat[3][2]) + (mat[1][2]); m_Frustum[BOTTOM][D] = (mat[3][3]) + (mat[1][3]); // Calculate the FRONT side (column3 + column4) m_Frustum[FRONT][A] = (mat[3][0]) + (mat[2][0]); m_Frustum[FRONT][B] = (mat[3][1]) + (mat[2][1]); m_Frustum[FRONT][C] = (mat[3][2]) + (mat[2][2]); m_Frustum[FRONT][D] = (mat[3][3]) + (mat[2][3]); // Calculate the BACK side (-column3 + column4) m_Frustum[BACK][A] = (mat[3][0]) - (mat[2][0]); m_Frustum[BACK][B] = (mat[3][1]) - (mat[2][1]); m_Frustum[BACK][C] = (mat[3][2]) - (mat[2][2]); m_Frustum[BACK][D] = (mat[3][3]) - (mat[2][3]);