QT + OpenCV的多线程

Question

I'm trying to code a simple program that reads three video files (actually, 3 cameras that are in the same room), using 3 different threads. The code I'm using is the following:

mainwindow.cpp

void MainWindow::init()
{
    numCams = 3;

    // Resize the video for displaying to the size of the widget
    int WidgetHeight = ui->CVWidget1->height();
    int WidgetWidth  = ui->CVWidget1->width();

    for (int i = 0; i < numCams; i++){
        // Create threads
        threads[i] = new QThread;

        // Create workers
        string Path = "/Users/alex/Desktop/PruebasHilos/Videos/" + to_string(i+1) + ".m2v";
        workers[i] = new Worker(QString::fromStdString(Path), i, WidgetHeight, WidgetWidth);

        workers[i]->moveToThread(threads[i]);

        connectSignals2Slots(threads[i], workers[i]);

        threads[i]->start();
        qDebug() << "Thread from camera " << (i+1) << " started";
    }
}

void MainWindow::connectSignals2Slots(QThread *thread, Worker *worker)
{
    connect(thread, SIGNAL(started()), worker, SLOT(readVideo()));
    connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    connect(worker, SIGNAL(frameFinished(Mat, int)), this, SLOT(displayFrame(Mat,int)));
    connect(worker, SIGNAL(finished(int)), thread, SLOT(quit()));
    connect(worker, SIGNAL(finished(int)), worker, SLOT(deleteLater()));
}

void MainWindow::displayFrame(Mat frame, int index)
{
    if (index == 0) {
        // Camera 1
        ui->CVWidget1->showImage(frame);
    }
    else if (index == 1) {
        // Camera 2
        ui->CVWidget2->showImage(frame);
    }
    else if (index == 2) {
        // Camera 3
        ui->CVWidget3->showImage(frame);
    }
}

worker.cpp

Worker::Worker(QString path, int id, int WidgetHeight, int WidgetWidth) : filepath(path), index(id), WidgetHeight(WidgetHeight), WidgetWidth(WidgetWidth) {
}

Worker::~Worker(){
}

void Worker::readVideo()
{
    VideoCapture cap(filepath.toStdString());

    if (! cap.isOpened()) {
        qDebug() << "Can't open video file " << filepath;
        emit finished(index);
        return;
    }

    Mat ActualFrame;
    while (true) {
        cap >> ActualFrame;

        if (ActualFrame.empty()) {
            // Empty frame to display when the video has finished
            ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
            emit frameFinished(ActualFrame, index);

            qDebug() << "Video finished";
            break;
        }

        // Background Subtraction
        BackgroundSubtraction(ActualFrame, BackgroundMask);

        emit frameFinished(ActualFrame.clone(), index);
        QThread::msleep(35);
    }
    emit finished(index);
}

void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
    pMOG2->apply(ActualFrame, BackgroundMask);
}

Just reading the frames from VideoCapture and displaying them into the UI by another different class that uses QWidgets works well.
However, when I include the BackgroundSubstraction method, the UI does not display the same frame number for the three cameras, maybe Camera1 is computing frame 100 and Camera2 and Camera3 are in frame 110.
This is because some frames are calculated faster than other and this leads to syntonization problems.
I'm quite new using threads in QT so i would like to make some synconization between threads so I know when the three different frames have been process in order to call the displayFrame method, and so, that the three same frames are displayed at the exact same time.

EDIT:
I assume that the easiest way to do this is using Barriers.
http://www.boost.org/doc/libs/1_55_0/doc/html/thread/synchronization.html#thread.synchronization.barriers . But I have no clue how to do this.

EDIT 2: I have implemented this Syncronizacion using barriers and now the code looks like this:

barrier.h

#ifndef BARRIER_H
#define BARRIER_H

#include <QMutex>
#include <QWaitCondition>
#include <QSharedPointer>

// Data "pimpl" class (not to be used directly)
class BarrierData
{
public:
    BarrierData(int count) : count(count) {}

    void wait() {
        mutex.lock();
        --count;
        if (count > 0)
            condition.wait(&mutex);
        else
            condition.wakeAll();
        mutex.unlock();
    }
private:
    Q_DISABLE_COPY(BarrierData)
    int count;
    QMutex mutex;
    QWaitCondition condition;
};

class Barrier {
public:
    // Create a barrier that will wait for count threads
    Barrier(int count) : d(new BarrierData(count)) {}
    void wait() {
        d->wait();
    }

private:
    QSharedPointer<BarrierData> d;
};

#endif // BARRIER_H

updated worker.cpp

void Worker::readVideo()
{
    VideoCapture cap(filepath.toStdString());

    int framenumber = 0;
    if (! cap.isOpened()) {
        qDebug() << "Can't open video file " << filepath;
        emit finished(index);
        return;
    }

    Mat ActualFrame;
    while (true) {
        cap >> ActualFrame;

        if (ActualFrame.empty()) {
            // Empty frame to display when the video has finished
            ActualFrame = Mat(Size(720, 576), CV_8UC3, Scalar(192, 0, 0));
            emit frameFinished(ActualFrame, index);

            qDebug() << "Video finished";
            break;
        }

        // Background Subtraction
        BackgroundSubtraction(ActualFrame, BackgroundMask);

        QThread::msleep(5);
        barrier.wait();
        qDebug() << "Thread " << index << " processing frame " << framenumber ;
        emit frameFinished(ActualFrame.clone(), index);
        framenumber++;
    }
    emit finished(index);
}

void Worker::BackgroundSubtraction(Mat ActualFrame, Mat &BackgroundMask)
{
    pMOG2->apply(ActualFrame, BackgroundMask);
}

It seems to work perfectly, however the output of the program is the following:

Thread  1  processing frame  0
Thread  0  processing frame  0
Thread  2  processing frame  0
Thread  2  processing frame  1
Thread  1  processing frame  1
Thread  0  processing frame  1
Thread  2  processing frame  2
Thread  1  processing frame  2
Thread  0  processing frame  2
Thread  2  processing frame  3
Thread  1  processing frame  3
Thread  0  processing frame  3
Thread  2  processing frame  4
Thread  1  processing frame  4
Thread  0  processing frame  4
Thread  2  processing frame  5
Thread  0  processing frame  5
Thread  1  processing frame  5
Thread  2  processing frame  6
Thread  1  processing frame  6
Thread  2  processing frame  7
Thread  0  processing frame  6
Thread  1  processing frame  7
Thread  2  processing frame  8
Thread  0  processing frame  7
Thread  1  processing frame  8
Thread  2  processing frame  9
Thread  0  processing frame  8
Thread  1  processing frame  9
Thread  1  processing frame  10
Thread  2  processing frame  10
Thread  0  processing frame  9
Thread  1  processing frame  11
Thread  2  processing frame  11
Thread  0  processing frame  10
Thread  1  processing frame  12

At the beginning the syncronization is perfectly working, but then it seems that the barrier is not working and threads are not waiting to each other...

EDIT 3: SOLVED It seems that changing the value of

QThread::msleep(5);

to

QThread::msleep(35);

solves the problem of the syncronization although I do not really understand the reason.

Answer 1

even without the background subtraction you'd need some synchronization to be sure that the same frame number is processed by each thread.

In Qt the easiest (and imho the right) way to do it is to remove the infinite loop and instead call a slot of each thread to compute the next image, after all the threads emitted their signal frameFinished.

You could further use some buffering to precompute images in your threads and just load them from that buffer. In that scenario you could do the following:

1. each of your threads fills his buffer in an endless loop as long as there is free buffer space available. If the buffer is full, the thread waits until buffer space got freed.

2. when your gui has displayed and has waited some time, it sends a signal that is connected to each thread's slot like sendMeANewImage.

3. each thread sends the next available image from its buffer, or waits (infinite loop or conditional wait) for an image, if the buffer is empty. Then emits a frameFinished signal and frees the used buffer-space.

4. when each thread has emitted the signal, display all the images, wait some time and emit sendMeANewImage again.

This isn't threadsafe yet, you'll have the critical sections in reading and writing from the buffer. For each buffer, create a QMutex and call mutex.lock() whenever reading or writing or asking size etc. from that buffer. Call mutex.unlock() immediately afterwards.

When a mutex is locked and another thread (or even the same thread) tries to lock it again, the thread will wait there, until the other thread has unlocked the mutex. That way, only a single thread can enter a critical section.