SDL2 and OpenGL stencil buffer don't work

Question

I'm a programing a game in c++ and sdl, opengl, but the stencil buffer seems not run. I have enabled stencil bits with this parameter:

SDL_GL_SetAttribute(SDL_GL_STENCIL_SIZE, 1);

Here is the code:

glColorMask(0,0,0,0);

_stencil.EnableStencil();
_stencil.StencilFunc(GL_ALWAYS,2,1);
_stencil.StencilOp(GL_REPLACE,GL_REPLACE,GL_REPLACE);

glBegin(GL_QUADS);
    glColor3f(0,0,0);
    glVertex2f(_size.x,_size.y);
    glVertex2f(_size.x,_size.y + _size.h);
    glVertex2f(_size.x + _size.w,_size.y + _size.h);
    glVertex2f(_size.x + _size.w,_size.y);
    glColor3f(1,1,1);
glEnd();

glColorMask(1,1,1,1);

_stencil.StencilFunc(GL_EQUAL,2,1);
_stencil.StencilOp(GL_KEEP,GL_KEEP,GL_KEEP);

for(int i = 0; i < _list.size(); i++)
    {

        _name.SetFontData(_fdata);
        _name.SetLabelText(_list[i].name);
        _name.SetLabelColor(255,255,255,255);

        _icon.BindTexture( _list[i].data.ID, _list[i].data.width , _list[i].data.height );

        /*if( _offsety + (_listGui.GetTextureInfo().height*i) + (_spacing_y*i) >= _size.y && _offsety + ((_listGui.GetTextureInfo().height*i)+_listGui.GetTextureInfo().height) + (_spacing_y*i) <= (_size.y + _size.h) )
        {*/
            if(i == 0)
            {
                _listGui.Render( _size.x + _spacing_x, _offsety );
                _name.RenderText(_size.x + _spacing_x + _label.x,_offsety + _label.y );
                _icon.Render( _size.x + (_iconPos.y + _spacing_x) , _offsety + _iconPos.y );
            }
            else
            {
                _listGui.Render( _size.x + _spacing_x, ( _offsety + ((_listGui.GetTextureInfo().height*i) + (_spacing_y*i) ) ) );

                _name.RenderText(
                        _size.x + ( _label.x + _spacing_x), 
                        _offsety + ( (_listGui.GetTextureInfo().height*i) + (_spacing_y*i) ) + _label.y 
                    );

                _icon.Render( 
                    _size.x + (_iconPos.x + _spacing_x),
                    _offsety + _iconPos.y + (_listGui.GetTextureInfo().height*i) + (_spacing_y*i) 
                    );

            }
        /*}*/
    }_stencil.DisableStencil();

_stencil.EnableStencil() is equal to glEnable(GL_STENCIL_TEST); same for _stencil.DisableStecil();

Answer 1

There is a problem with the mask value you use in your stencil state:

_stencil.StencilFunc(GL_ALWAYS,2,1);

The mask value, which is given by the 3rd argument of glStencilFunc() , is ANDed with the reference value and the value in the stencil buffer before any of the stenciling logic is applied. In the language of the spec:

The s least significant bits of mask are bitwise ANDed with both the reference and the stored stencil value, and the resulting masked values are those that participate in the comparison controlled by func.

With the values you use, the mask value 1 will be ANDed with the reference value 2, which results in an effective reference value of 0. In binary:

2     = 00000010
1     = 00000001
2 & 1 = 00000000

This means that in the fist pass, the stencil values of everything you render are set to 0, which is the value you already cleared them to. In the second pass, the values are compared with 0, and all pixels will pass the stencil test.

To get this working, you need to either change the reference value or the mask so that the reference value isn't masked out. With a reference value of 2, the mask could be something like 2 or 3. But if you only want to use one bit of stencil, the easiest way is to use 1 for the reference value.

This means that you would change the setup for the first pass to:

_stencil.EnableStencil();
_stencil.StencilFunc(GL_ALWAYS, 1, 1);
_stencil.StencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);

and for the second pass to:

_stencil.StencilFunc(GL_EQUAL, 1, 1);
_stencil.StencilOp(GL_KEEP, GL_KEEP, GL_KEEP);