Efficient way to read and process video files with Qt and OpenCV

Question

I have a few vision algorithms which perform well enough on live camera streams; however, it is far from good when I run these algorithms on video files; the stream slows down way too much, although it is fine when not running on a vision algorithm, which are executed by calling VideoAlgoVision->execute( grabbedFrame, &CurrentROI );

Here is how I read video so far:

//////////////////////////////////////////////////////////////////

void VisionUnit_widget::Play()
{
    if(IsVideoPause)
    {
        ui->pausePushButton->setEnabled(true);
        ui->playPushButton->setEnabled(false);
        IsVideoPause = false;
        TimerOpen->start( (int) (1000/FPS_open) );

        return;
    }
    else
    {
        ui->pausePushButton->setEnabled(true);
        ui->stopPushButton ->setEnabled(true);
        ui->rewPushButton  ->setEnabled(true);
        ui->ffdPushButton  ->setEnabled(true);
        ui->videoSlider    ->setEnabled(true);

        ui->playPushButton ->setEnabled(false);

        if(!VideoCap)
            VideoCap = new VideoCapture( FileName.toStdString() );
        else
            VideoCap->open( FileName.toStdString() );

        if( VideoCap->isOpened() )
        {
            FrameH    = (int) VideoCap->get(CV_CAP_PROP_FRAME_HEIGHT);
            FrameW    = (int) VideoCap->get(CV_CAP_PROP_FRAME_WIDTH);
            FPS_open  = (int) VideoCap->get(CV_CAP_PROP_FPS);
            NumFrames = (int) VideoCap->get(CV_CAP_PROP_FRAME_COUNT);

            ui->videoSlider->setMaximum( (int)NumFrames );

            ui->videoSlider->setEnabled(true);
            READCOUNT = 0;
            TimerOpen->start( (int) (1000/FPS_open) );
        }
    }    
}


//////////////////////////////////////////////////////////////////

void VisionUnit_widget::Pause()
{
    ui->playPushButton->setEnabled(true);
    ui->pausePushButton->setEnabled(false);
    TimerOpen->stop();
    IsVideoPause = true;
}


//////////////////////////////////////////////////////////////////

void VisionUnit_widget::Stop()
{
    ui->stopPushButton->setEnabled(false);
    ui->playPushButton->setEnabled(false);
    ui->pausePushButton->setEnabled(false);
    ui->rewPushButton->setEnabled(false);
    ui->ffdPushButton->setEnabled(false);

    FileName = "";

    TimerOpen->stop();

    READCOUNT = 0;

    ui->videoSlider->setSliderPosition(0);
    ui->videoSlider->setEnabled(false);

    ui->frameLabel->setText( "No camera connected" );

    delete TimerOpen;
    TimerOpen = 0;

    if(VideoCap)
    {
        delete VideoCap;
        VideoCap = 0;
    }

    if(VideoAlgoVision)
    {
        delete VideoAlgoVision;
        VideoAlgoVision = 0;
    }
}


//////////////////////////////////////////////////////////////////

void VisionUnit_widget::Read()
{
    READCOUNT++;

    // Update Video Player's slider
    ui->videoSlider->setValue(READCOUNT);

    if(READCOUNT >= NumFrames) // if avi ends
    {
        Pause();
        return;
    }

    Mat grabbedFrame;


    // Get next frame
    if(VideoCap->isOpened() && VideoCap->read(grabbedFrame))
    {
        // Execute the vision filter
        if(VideoAlgoVision)
            VideoAlgoVision->execute( grabbedFrame, &CurrentROI );

        // Convert Mat to QImage
        QImage frame = MatToQImage( grabbedFrame );

        // Update the display
        UpdateFrame( frame );
    }
}


//////////////////////////////////////////////////////////////////

QImage VisionUnit_widget::MatToQImage(const Mat& mat)
{
    // 8-bits unsigned, NO. OF CHANNELS = 1
    if(mat.type()==CV_8UC1)
    {
        // Set the color table (used to translate colour indexes to qRgb values)
        QVector<QRgb> colorTable;

        for (int i=0; i<256; i++)
        {
            colorTable.push_back(qRgb(i,i,i));
        }

        // Copy input Mat
        const uchar *qImageBuffer = (const uchar*)mat.data;

        // Create QImage with same dimensions as input Mat
        QImage img(qImageBuffer, mat.cols, mat.rows, mat.step, QImage::Format_Indexed8);

        img.setColorTable(colorTable);
        return img;
    }
    // 8-bits unsigned, NO. OF CHANNELS=3
    else if(mat.type()==CV_8UC3)
    {
        // Copy input Mat
        const uchar *qImageBuffer = (const uchar*)mat.data;

        // Create QImage with same dimensions as input Mat
        QImage img(qImageBuffer, mat.cols, mat.rows, mat.step, QImage::Format_RGB888);

        return img.rgbSwapped();
    }
    else
    {
        return QImage();
    }
}

So, my question is, is there a better way than this to read video files with Qt and OpenCV, including video processing? Should I adapt the QTimer timing at run time? I start it with TimerOpen->start( (int) (1000/FPS_open) ); but obviously the vision algorithm will slow down the whole thing. Any thought?

There may be some optimization to make on the vision algorithm, however my point here is they do well on my webcam and IP cameras, having me think there is something wrong with the way I read/use video files.

thx

Answer 1

You did not provide the whole code however i guess that you have connected TimerOpen 's timeout() signal to VisionUnit_widget::Read() slot. If this is the case, you are accumulating 1000/FPS_open and processing time.

Changing your desing to something like the following will solve it :

int fSpace = 1000 / FPS;
QTime timer;
timer.start();

forever {
    if(timer.elapsed() < fSpace)
        msleep(1);

    timer.restart();

    ..Process the frame..
}

And probably it is better to move this to another thread from main thread.