QT5 with OpenGL and Kinect v2: wrong image presentation

Question

I would like to show a color image from the kinect v2 sensor over openGl in a Qt widget. The problem is, that the presentation of this image is incorrect.

Here is the code where I get the color frame:

class:

cv::Mat                     ColorMap;
std::vector<BYTE>           colorBuffer;
int                         color_width;
int                         color_height;

code:

if(colorBuffer.empty())
   colorBuffer.resize(color_height * color_width * 4 * sizeof(unsigned char));

hr = ColorFrame->CopyConvertedFrameDataToArray((UINT)colorBuffer.size(),
                             &colorBuffer[0], ColorImageFormat::ColorImageFormat_Bgra );

ColorMap = cv::Mat( color_height, color_width, CV_8UC4, &colorBuffer[0]);

That means, I get the color information as BGR Format with an alpha channel and copy it to a matrix with 4 channels, (BRGA) and each channel has 8Bit from 0 to 255. Correct?

In the next step, I resize it as the same size as the widget:

    Kinect->CreateFrame();
    cv::Mat GUIColorImage;
    Kinect->ColorMap.copyTo(GUIColorImage);
    cv::resize(GUIColorImage,GUIColorImage,
    cv::Size(ui->Left_Widget_Color->width(),ui->Left_Widget_Color->height()));

Than I've tried two convert methods: 1. convert to BGR 2. convert to 8UC3 (8-Bit, unsigned char, 3 channels (the same as BGR?))

1: GUIColorImage.convertTo(GUIColorImage,CV_BGRA2BGR);

2: GUIColorImage.convertTo(GUIColorImage,CV_8UC3);

But no solutions works.

After the conversion, I try to display it over openGl with:

LeftWidgetColor.UpdateImage(GUIColorImage);

LeftWidgetColor is a QOpenGLWidget:

header:

class RenderWidget : public QOpenGLWidget
{
    Q_OBJECT
public:
    RenderWidget(QWidget *parent);
    ~RenderWidget();
    void UpdateImage(cv::Mat newimage);
    void initializeGL();
    void paintGL();

private:
    int                 width;
    int                 height;
    GLuint              texture;
    cv::Mat             image;

signals:
    void info(QString msg);
    void error(QString msg);
    void DepthValueAt(cv::Point2i DepthPosition);

protected:
    void mousePressEvent(QMouseEvent *event);
    void mouseReleaseEvent(QMouseEvent *event);
    void mouseMoveEvent(QMouseEvent *event);
};

code:

RenderWidget::RenderWidget(QWidget *parent) : QOpenGLWidget(parent)
{
    width = this->size().width();
    height = this->size().height();
    texture = 0;
    initializeGL();
}

RenderWidget::~RenderWidget()
{
    glDeleteTextures(1, &texture);
}


void RenderWidget::initializeGL()
{
    //Background Color is black
    glClearColor(0,0,0,1);

    //Storage of Pixelmode for two-dimensional textures
    glPixelStorei (GL_UNPACK_ALIGNMENT, 1);

    //Create texture
    glGenTextures(1, &texture);

    glViewport(0, 0, width, height);


}

void RenderWidget::paintGL()
{
    // Clear the screen and depth buffer (with black)
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    // Select the model view matrix and reset it
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

    glTranslatef(0.0f, 0.0f, 0.0f);

    // Abort drawing if OpenCV was unable to open the camera
    if (image.empty())
    {
        return;
    }
    glEnable(GL_TEXTURE_RECTANGLE_ARB);

    // Typical texture generation using data from the bitmap
    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, texture);

    // Transfer image data to the GPU
    glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0,
                 3, image.cols, image.rows, 0,
                 GL_BGR, GL_UNSIGNED_BYTE, image.data);


    if (glGetError() != GL_NO_ERROR)
    {
    }

    // Draw a 2D face with texture
    glBegin(GL_QUADS);
        glTexCoord2f(0, 0);                   glVertex2f(1, 1);
        glTexCoord2f(image.cols, 0);          glVertex2f(-1, 1);
        glTexCoord2f(image.cols, image.rows); glVertex2f(-1, -1);
        glTexCoord2f(0, image.rows);          glVertex2f(1, -1);
    glEnd();

glDisable(GL_TEXTURE_RECTANGLE_ARB);
}

void RenderWidget::UpdateImage(cv::Mat newimage)
{
    newimage.copyTo(image);
    update();
}

I guess the problem is in:

glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0,
             3, image.cols, image.rows, 0,
             GL_BGR, GL_UNSIGNED_BYTE, image.data);

But I can't find it. I've declared 3-channels, BGR, 8-Bit = 1 Byte as unsigned. Does someone know where the mistake is?

If i show it over imshow (openCv class), it works fine.

Answer 1

My Mistake was, that I used cv::Mat::convertTo(): but here they wrote, that convertTo() should not used to convert the format (Only to change the data type). cv::cvtColor is the right function and it works well for me. So I've changed the conversion from:

GUIColorImage.convertTo(GUIColorImage,CV_BGRA2BGR);

to

cv::cvtColor(GUIColorImage,GUIColorImage,CV_BGRA2BGR);