using opencv waitKey() in a multithreading application

Question

I have a multi-threading application written in C++ with Qt5.7 and OpenNI. It has a main thread that starts a second thread which capture frame from a .oni recording file (asus xtion pro live) does some processing and through the Qt signal-slot mechanism pass the frame to the main thread, which display it using imshow() . What I want to do is to implement a pause key, so pressing for example 'p' the processing pause. I am thinking of something like this:

void Camera::run(){
  while(!cameraStop && this->device.isValid())
    {
      try {
        if (!buttonPause) {
            getFrame();
            process();
            emit sigFrameImageReady(frame);
            if (cv::waitKey(1)==112){
              setButtonPause(!(getButtonPause()));
            }
          }

      }
      catch(std::exception &ex) {
        std::cerr << "getFrame()" << ex.what() << std::endl;
      }
    }
}

In this way it doesn't work, I think that's because the frame is displayed by another thread (the main one), the waitKey() here simply blocks the entire process, but if I put it in the main thread, just after imshow() in this way:

void Process::FrameImageReady(cv::Mat FrameImage)
{
  if (modedebug)
    cv::imshow("bgr", FrameImage);
  if (cv::waitKey(1)==112){
    cam->setButtonPause(!(getButtonPause()));
  }
} 

waitkey seems to be ignored (image displaying works fine).. any idea?

EDIT The GUI part is only for debugging purpose.

Answer 1

You should implement thread safe FIFO bufer or circular buffer in your displaying thread. Signal from the camera thread would be pushing images to this buffer and the displaying thread would be taking them out and display them in a separate loop. Only that way you separate the camera event loop from the display thread.

Answer 2

Maybe it's a late answer but I would like to present what I have done in my recent project, in case anyone is in the similar situation and want to use the same techniques.

Basically my project is a command line program and uses QT for multi threading & inter-thread communications, while we also want some minimal UI to display images, and handle user key events, so OpenCV Highgui module is pretty much enough for me. And here is what I did to make them work together:

1. (Important) you have to build OpenCV with QT support to make the following code work, that is, check WITH_QT option in cmake-gui when building OpenCV. Or if you use vcpkg, use:

    vcpkg install opencv[qt]:x64-windows 

2. First create the class for working thread, I use this thread to constantly retrieve image frames from the camera buffer and notify the main thread every time the image is ready:

    class MyThread : public QThread { Q_OBJECT public: MyThread() : mAbort(false) { } ~MyThread() { mMutex.lock(); mAbort = true; // make the thread out of the loop mMutex.unlock(); wait(); // wait until the run() function returns cout << "thread terminated" << endl; } protected: void run() override { while(!mAbort) { cv::Mat frame; if (myCamera->grab(frame) && !frame.empty()) { emit imageReady(frame); } else { cout << "failed to grab" << endl; } } } signals: void imageReady(const cv::Mat& frame); private: bool mAbort; QMutex mMutex; }; 

3. Then create the controller class to handle the signal from working thread, I use it to display the frame. Note it is running in the main thread:

    class MyController : public QObject { Q_OBJECT public: MyController(MyThread* pThread) : mThread(pThread) { // this is needed as the argument is passed through queued connection qRegisterMetaType<cv::Mat>("CvMat"); connect(mThread, &MyThread::imageReady, this, &MyController::onImageReady); connect(mThread, &MyThread::finished, mThread, &QObject::deleteLater); mThread->start(); } public slots: void onImageReady(const cv::Mat& frame) { cv::imshow("camera live view", frame); } private: MyThread* mThread; }; 

4. Now in the main function, we can start the thread, and handle key events using cv::waitKey() , note it will also start the QT main event loop internally.

    Q_DECLARE_METATYPE(cv::Mat) ... int main(int argc, char* argv[]) { ... // start working thread MyThread thread = new MyThread(); MyController controller(thread); // handle key events while (true) { int key = cv::waitKey(); cout << "key = " << key << endl; // press "ESC" to exit if (key == 27) { break; } } // clean up ... delete thread; delete myCamera; return 0; }