Draw trajectories of objects in a video using annotations! [OpenCV]

Question

I have several videos about 5 minutes each along with annotation data containing information about each object's bounding box coordinates at each frame number. I am trying to read the videos and draw lines between center of bounding boxes frame by frame (when the current frame number matches the number from the ground truth data). I don't want to do this in a batch process, but every 30 or 60 frames would work for me. Here is my code:

VideoCapture capture(path_video);
if (!capture.isOpened()){
   cout <<  "Failed to capture frame/Open the file" << "\n";
   return 0;
}
bool stop(false);
while(!stop){
   capture >> frame;
   if (frame.data==NULL) {
      break;
   }
   double rate = capture.get(CV_CAP_PROP_FPS);
   int delay = 1000/rate;
   frmNum = (capture.get(CV_CAP_PROP_POS_FRAMES));
   for (int i=0 ; i<db.size() ; i++){//db is a vector of vector that has annotation data for each object splited in inner vectors , and is sorted ascendingly on start frame number of objects.
      if (!db[i].empty()){
         for (int j=1 ; j<db[i].size() ; j++){
            if(frmNum == db[i][j-1].currFrame){
               cv::line(frame, db[i][j-1].pnt, db[i][j].pnt,Scalar(255,0,0),2);
            }
            else{
               break;
            }
         }
      }
   }
   imshow("Video", frame);
   int key = waitKey(delay);
   if (key==27){
      break;
   }

I checked and my if condition becomes true but no line is drawn on the video. I guess I don't see the lines because frames are changing and the drawn lines are cleared by new frames, But I couldnt come up with an alternative way. Thanks for your help.

Answer 1

If you want to draw the annotations on every frame and you don't mind that the information may be "obsolete" in some cases, I would do the following:

• have a global image variable called "overlay" which would hold the most up to date (in regards to current frame number) representation of annotations and would have the same size as a single frame from the videostream
• maintain an array of indices called "last_object_annotation_idx", which would store for each object the last index of its annotation already seen/used (initially set to -1 for each object)
• in each iteration of the main loop, update the "last_object_annotation_idx" array (that is, for each object check if the current frame number matches the "currFrame" field of the next annotation - I am using the information that the array of annotations is sorted)
• if there was a change to the array, redraw the overlay image using annotations referenced from "last_object_annotation_idx"
• finally, add the overlay image to the frame and display the result.

Also, in the if statement

if(frmNum == db[i][j-1].currFrame){
    cv::line(frame, db[i][j-1].pnt, db[i][j].pnt,Scalar(255,0,0),2);
}
else{
    break;
}

isn't the "break" kind of wrong? It means you will break out of the loop if the first check fails, so you will not see anything past the first index in that case.