OpenCV C++ Multi-threading to Improve Framerate

Question

I have an opencv program where the image processing is sensitive to having a stable and relatively high framerate in the video capture. Unfortunately, all the image processing I do is slowing down the framerate significantly, resulting in erroneous behavior in my program. I believe that putting the camera on a separate thread and having the image processing happen on its own thread would improve framerate, but I am not sure how to do this. Can anyone guide me through the process?

UPDATE: After doing some research on threads, I managed to implement threading to where the final video feed post-processed is displayed. However, somehow my implementation of it is causing the image processing methods to fail (ex. before I could successfully track a moving object, now it is erroneous whether or not it is tracked). I suspect this has something to do with the image processing algorithms not being able to process each frame fast enough as new frames are read in. How can I improve this implementation so that my processing methods worked as they did without multithreading?

void CaptureFrames() {
  VideoCapture capture(0);
  if (!capture.isOpened()) {
      cout << "cannot open camera" << endl;
  }

  while (true) {
      //CAMERAFRAME is a global Mat defined at the top of my program
      capture.read(CAMERAFRAME);
      if (waitKey(30) >= 0) { break; }
  }
}

void ProcessFrames() {

  while (true) {



    Mat hsvFrame;
    Mat binMat;
    Mat grayFrame;
    Mat grayBinMat;

    if (!CAMERAFRAME.empty()) {
        //do image processing functions (these worked before implementing threads and are not causing errors)

        imshow("gray binary", grayBinMat);
        imshow("multithread test", CAMERAFRAME);
    }



    if (waitKey(30) >= 0) { break; }
   }
}

int main(int argc, char** argv[]) {
  thread t1(CaptureFrames);
  thread t2(ProcessFrames);

  while(true) {
    if(waitKey(30) >= 0) {break;}
  }

  return 0;
}

Answer 1

Try the older version again but remove this last line from the ProcessFrames function.

if (waitKey(30) >= 0) { break; }

On showing images don't make it wait again for 30 m-seconds, the while loop will be enough