Self-occlusion-aware multiple geometry blending in C++ OpenGL (rendering)

Question

I have got several meshes (~100) of the same complex object in various poses with slightly different rotation and translation parameters. The object consists of multiple rigid components like arms and legs.

The goal is to generate a unique grayscale picture showing the accumulation of these poses for a particular body part. The heat-map obtained gives an idea of probable pixel locations for the body part, where white represents maximum probability, and black minimum (the lighter the higher probability). Say I'm interested in the accumulation of the legs. If many leg pose samples lie on the same (x,y) pixel location, than I expect to see light pixels there. Ultimately the leg poses might not exactly overlap, so I also expect to see a smooth transition to the black low probability around the leg silhouette boundaries.

To solve this task I have decided to use rendering in OpenGL frame buffers as these are known to be computationally cheap, and because I need to run this accumulation procedure very often.

What I did is the following. I accumulate the corresponding renderings of the body part I'm interested in (let's still keep the leg example) on the same frame buffer 'fboLegsId' using GL_BLEND. In order to discriminate between the legs and the rest of the body, I texture the mesh with two colors:

• rgba(gray,gray,gray,255) for the legs, where gray = 255 / Number of samples = 255/100

• rgba(0,0,0,0) for the rest of the body

Then I accumulate the 100 renderings (which for the leg should sum up to white = 255) by doing the following:

glBindFramebuffer(GL_FRAMEBUFFER, fboLegsId);

glClearColor(0,0,0,255);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glBlendFunc(GL_ONE, GL_ONE);
glEnable(GL_BLEND);

for each sample s = 0...100

   mesh.render(pose s);

end

glReadPixels(...)

This performs almost as I expected. I do obtain the smooth grayscale heat-map I wanted. However there are self-occlusion problems which arise even when I use only 1 sample. Say for a single pose sample, one of the arms moved before the leg, partially occluding them. I expect the influence of the occluded leg parts to be cancelled during rendering. However it renders as if the arm is invisible/translucent, allowing for pixels behind to be fully shown. This leads to wrong renderings and therefore wrong accumulations.

If I simple disable blending, I see the correct self-occlusion aware result. So, apparently the problem lies somewhere at blending time.

I also tried different blending functions, and so far the following one produced the closer results to a self-occlusion aware accumulation approach: glBlendFunc(GL_ONE, GL_SRC_ALPHA); Anyway there is still a problem here: one single sample looks now correct; two or more accumulated samples instead show overlapping artefacts with other samples. It looks like each accumulation replaces the current buffer pixel if the pixel is not part of the legs. And if the leg was found many times in front of the (let's say) the arm, than it becomes darker and darker, instead of lighter and lighter. I tried to fix this by clearing depth buffer at each rendering iteration enabling depth computations, but this did not solve the problem.

I feel like there is either something conceptually wrong in my approach, or a small mistake somewhere.

---

I've tried a different approach based on the suggestions which performs as expected. Now I'm working with 2 frame buffers. The first one (SingleFBO) is used to render single samples with correct self-occlusion handling. The second (AccFBO) is used to accumulate the 2D textures from the first buffer using blending. Please, check my code below:

 // clear the accumulation buffer
 glBindFramebuffer(GL_FRAMEBUFFER, AccFBO);
 glClearColor(0.f, 0.f, 0.f, 1.f);
 glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

 for each sample s = 0...100
 {
        // set rendering destination to SingleFBO
        glBindFramebuffer(GL_FRAMEBUFFER, SingleFBO);

        glClearColor(0.f, 0.f, 0.f, 1.f);
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        glEnable(GL_DEPTH_TEST);
        glDisable(GL_LIGHTING);
        mesh->render(pose s);
        glDisable(GL_DEPTH_TEST);
        glEnable(GL_LIGHTING);

        // set rendering destination to the accumulation buffer
        glBindFramebuffer(GL_FRAMEBUFFER, AccFBO);

        glClear(GL_DEPTH_BUFFER_BIT);

        glBlendFunc(GL_ONE, GL_ONE);
        glEnable(GL_BLEND);

        // draw texture from previous buffer to a quad
        glBindTexture(GL_TEXTURE_2D, textureLeg);
        glEnable(GL_TEXTURE_2D);

        glDisable(GL_DEPTH_TEST);
        glDisable(GL_LIGHTING);
        glDepthMask(GL_FALSE);

        glMatrixMode(GL_PROJECTION);
        glPushMatrix();
        glLoadIdentity();
        glMatrixMode(GL_MODELVIEW);
        glPushMatrix();
        glLoadIdentity();

        glBegin( GL_QUADS );
        {
            glTexCoord2f(0,0); glVertex2f(-1.0f, -1.0f);
            glTexCoord2f(1,0); glVertex2f(1.0f, -1.0f);
            glTexCoord2f(1,1); glVertex2f(1.0f, 1.0f);
            glTexCoord2f(0,1); glVertex2f(-1.0f, 1.0f);
        }
        glEnd();

        glPopMatrix();
        glMatrixMode(GL_PROJECTION);
        glPopMatrix();
        glMatrixMode(GL_MODELVIEW);

        // restore
        glDisable(GL_TEXTURE_2D);
        glEnable(GL_DEPTH_TEST);
        glEnable(GL_LIGHTING);
        glDepthMask(GL_TRUE);

        glDisable(GL_BLEND);
  }

  glBindFramebuffer(GL_FRAMEBUFFER, AccFBO);
  glReadPixels(...)

Please, check also my (standard) code for initializing the SingleFBO (similarly for AccFBO):

    // create a texture object
    glGenTextures(1, &textureLeg);
    glBindTexture(GL_TEXTURE_2D, textureLeg);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_GENERATE_MIPMAP, GL_TRUE);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, width, height, 0,
                 GL_RGB, GL_UNSIGNED_BYTE, 0);
    glBindTexture(GL_TEXTURE_2D, 0);

    // create a renderbuffer object to store depth info
    glGenRenderbuffers(1, &rboLeg);
    glBindRenderbuffer(GL_RENDERBUFFER, rboLeg);
    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT,
                          width, height);
    glBindRenderbuffer(GL_RENDERBUFFER, 0);

    // create a framebuffer object
    glGenFramebuffers(1, &SingleFBO);
    glBindFramebuffer(GL_FRAMEBUFFER, SingleFBO);

    // attach the texture to FBO color attachment point
    glFramebufferTexture2D(GL_FRAMEBUFFER,        // 1. fbo target: GL_FRAMEBUFFER 
                           GL_COLOR_ATTACHMENT0,  // 2. attachment point
                           GL_TEXTURE_2D,         // 3. tex target: GL_TEXTURE_2D
                           textureLeg,             // 4. tex ID
                           0);                    // 5. mipmap level: 0(base)

    // attach the renderbuffer to depth attachment point
    glFramebufferRenderbuffer(GL_FRAMEBUFFER,      // 1. fbo target: GL_FRAMEBUFFER
                              GL_DEPTH_ATTACHMENT, // 2. attachment point
                              GL_RENDERBUFFER,     // 3. rbo target: GL_RENDERBUFFER
                              rboLeg);              // 4. rbo ID

    // check FBO status
    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    if(status != GL_FRAMEBUFFER_COMPLETE)
        error(...);

    // switch back to window-system-provided framebuffer
    glBindFramebuffer(GL_FRAMEBUFFER, 0);

Answer 1

Here's a different approach:

Create two frame buffers: normal and acc . normal frame buffer should have a texture storage (with glFramebufferTexture2D ).

Here's the basic algorithm:

1. Clear acc to black
2. Bind normal , clear to black, and render scene with white legs, and other parts black
3. Bind acc , render a full screen rectangle, with normal texture on it, with blend mode GL_ONE, GL_ONE
4. Forward the animation, and if it haven't finished, goto 2.
5. You have the result in acc

So, basically, acc will contain the individual frames summed.