Do not work stencil in Java Android Opengl es 2.0

Question

In android opengl es 2.0 I'm drawing scene by standard renderer (implements GLSurfaceView.Renderer)

@Override
public void onDrawFrame(GL10 gl10)
{
        GLES20.glEnable(GLES20.GL_STENCIL_TEST);
        GLES20.glClearStencil(1);
        GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);
        GLES20.glEnable(GLES20.GL_DEPTH_TEST);


        glStencilMask(0x00);

        shaderTexture2.useProgram();

        GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
        GLES20. glStencilFunc(GLES20.GL_ALWAYS, 1, 0xFF);
        GLES20.glStencilMask(0xFF);

        //1 step. draw red triangle
        shaderTexture2.draw(buf_slim, mvp, mLightPosInEyeSpace, tex1);


        GLES20.glStencilFunc(GLES20.GL_NOTEQUAL, 1, 0xFF);
        GLES20.glStencilMask(0x00);

        //2 step. draw blue triangle
        shaderTexture2.draw(buf_big, mvp, mLightPosInEyeSpace, tex1);

        GLES20.glStencilMask(0xFF);

}

If in first line of method turn off stencil (for examle comment GLES20.glEnable(GLES20.GL_STENCIL_TEST); ) then both triangles draws( first pic).If stencil turned on then I'm expect at step 1 that at drawing red triangle stencil buffer will be filled with ones, and at step 2 blue triangle will be cuted only in area where was ones from first step (second pic). Instead when stencil turned on simply drawing full red triangle, without any blue triangle (third pic). When and which funcs and masks of stencil I should use to draw as i expect?

Answer 1

Considering specific statements in the order they are executed...

The following code will set every pixel in the stencil buffer to 1.

GLES20.glClearStencil(1);
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);

Next the red triangle is rendered. But the stencil function is setup such that any time the stencil and depth tests pass the value 1 will be written to the stencil buffer. That doesn't change anything because the stencil buffer is already filled with 1.

GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
GLES20.glStencilFunc(GLES20.GL_ALWAYS, 1, 0xFF);
GLES20.glStencilMask(0xFF);

// 1 step. draw red triangle
shaderTexture2.draw(buf_slim, mvp, mLightPosInEyeSpace, tex1);

At this point the stencil buffer is still filled with the value 1.

Now the stencil function is setup to pass when the current value is not equal to 1. Hence the stencil test will always fail and no fragments of the blue triangle will be visible.

GLES20.glStencilFunc(GLES20.GL_NOTEQUAL, 1, 0xFF);
GLES20.glStencilMask(0x00);

// 2 step. draw blue triangle
shaderTexture2.draw(buf_big, mvp, mLightPosInEyeSpace, tex1);

The following code (untested) alters things such that 0 is used as the stencil clear value. The value 1 is written to those stencil pixels covered by the red triangle. Finally the blue triangle is rendered only to those pixels that have a corresponding stencil pixel equal to 1.

@Override
public void onDrawFrame (GL10 gl10)
{
    GLES20.glEnable(GLES20.GL_STENCIL_TEST);
    GLES20.glClearStencil(0);
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);
    GLES20.glEnable(GLES20.GL_DEPTH_TEST);

    glStencilMask(0x00);

    shaderTexture2.useProgram();

    GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
    GLES20.glStencilFunc(GLES20.GL_ALWAYS, 1, 0xFF);
    GLES20.glStencilMask(0xFF);

    // 1 step. draw red triangle
    shaderTexture2.draw(buf_slim, mvp, mLightPosInEyeSpace, tex1);

    GLES20.glStencilFunc(GLES20.GL_EQUAL, 1, 0xFF);
    GLES20.glStencilMask(0x00);

    // 2 step. draw blue triangle
    shaderTexture2.draw(buf_big, mvp, mLightPosInEyeSpace, tex1);

    GLES20.glStencilMask(0xFF);
}