how to capture images per second from video files by using opencv and c++?

Question

I have encountered on designing program to allow capturing images every second from video files (avi, mp4, etc...).

First, I was able to capture images frame by frame from video file. Second, I was able to analyze pixel color values from images in the same folder at the same time and saved pixel value in the txt file.

And here I have some problem. I am now trying to combine these two codes at once, but I have strange results. I refer the code below.

int main(){
VideoCapture cap("D:\\data\\extra\\video200ul.avi"); 
if (!cap.isOpened())  
    return -1;

Ptr<BackgroundSubtractor> pMOG2 = createBackgroundSubtractorMOG2(20, 16, true);

Mat fg_mask;
Mat frame;
int count = 0;

String name, folder;

for (;;) {
    // Get frame
    cap >> frame; // get a new frame from video
    ++count;
    // Update counter

    // Background subtraction
    if (count % 2 == 0) {
        pMOG2->apply(frame, fg_mask, 0.001);

        cout << count << endl;

        if (!frame.empty()) {
            imshow("frame", frame);
            //      imshow("fg_mask", fg_mask);
        }
        // Save foreground mask
        name = "mask" + std::to_string(count) + ".png";
        //      string name = "mask_" + std::to_string(static_cast<long long>(count) + ".png";
        folder = imwrite("D:\\data\\extra\\" + name, frame);

    }
    anal(folder);
}   
waitKey(0);
return 0;

}

First, The code above I wrote is for capturing images frame by frame from video file. However, if I got the images per frame, I will have so many pictures on my folder, so I would like to capture an image per second from the video file. I have tried to use CV_CAP_PROP_POS_MSEC instead using cap << frame, but it did not work for me.

Second, when I merge this code to another code what I wrote below, it showed some error messages like, "libpng warning image width, length, data are zero in ihdr."

int anal(String folder) {

folder = "D:\\data\\extra\\*.png"; 
vector<String> filenames;

glob(folder, filenames);

cv::Mat ori_image;

for (size_t i = 0; i < filenames.size(); ++i) {

    ori_image = imread(filenames[i], IMREAD_COLOR);

    if (ori_image.empty()) {
        cout << "Check your file again." << std::endl;
        return -1;
    }

    rectangle(ori_image, Point(215, 98), Point(245, 110), Scalar(0, 255, 255), 1); 

    imshow("Original Image", ori_image);

    cv::Scalar sums;
    sums = cv::sum(ori_image);

    double totalSum = sums[0] + sums[1] + sums[2];

    if (totalSum <= 0) {
        cout << "$$ RGB percentage $$" << " \n\n";
        cout << "R: " << 100.0 / 3 << " % \n";
        cout << "G: " << 100.0 / 3 << " % \n";
        cout << "B: " << 100.0 / 3 << " % \n\n";
    }
    else {
        cout << "$$ RGB percentage $$" << " \n\n"; // red value
        cout << "R: " << sums[2] / totalSum * 100 << " % \n"; // red value
        cout << "G: " << sums[1] / totalSum * 100 << " % \n"; // green value
        cout << "B: " << sums[0] / totalSum * 100 << " % \n\n"; // blue value
    }

}

as I prepared the code above, I tried to calculate red, blue, green percentages of all the captured images from the video. However, when I separate these two code and run them, they worked fine, but if I merge them together, It showed error messages.

I would like to combine these two code for analysis for color values from the captured images at video every second.

Please help me out this problem.

Thank you in advance.

-----------Edited part----------------------

I used your revised version and applied to my updated code,

void imageAnalysis(std::string folder, cv::Mat frame){
cv::Mat ori_image = frame.clone();
std::string path = folder;
cv::rectangle(ori_image, Point(215, 105), Point(245, 120), Scalar(0, 255, 255), 1);

cv::imshow("Original Image", ori_image);
cv::waitKey(1);


String folder = "D:\\data\\dfdf\\*.png";
vector<String> filenames;
cv::glob(path, filenames);

for (size_t t = 0; t < filenames.size(); t++) {
    ori_image = imread(filenames[t], IMREAD_COLOR); // ori_image

    if (ori_image.empty()) {    //ori_image
        cout << "Check your file again." << "\n";
        break;
        //return -1;
    }

    rectangle(ori_image, Point(215, 105), Point(245, 120), Scalar(0, 255, 255), 1); 
    imshow("Original Image", ori_image);
    cv::waitKey(1);

    Mat image_HSV;
    cvtColor(ori_image, image_HSV, CV_BGR2HSV);

    double h = 0.0;
    double s = 0.0;
    double v = 0.0;

    int col = image_HSV.cols; // 480
    int row = image_HSV.rows; // 272

    int corow = ((col - 235) - 215) * ((row - 152) - 108);  

    Mat mask;
    inRange(image_HSV, Scalar(100, 0, 0), Scalar(100, 255, 255), mask); // convert binary
    image_HSV.setTo(Scalar(0, 0, 0), mask);

    for (int i = 108; i < row - 152; i++) {     
        for (int j = 215; j < col - 235; j++) {
            Vec3b hsv = image_HSV.at<cv::Vec3b>(i, j);

            h += (int)(hsv.val[0]);
            s += (int)(hsv.val[1]);
            v += (int)(hsv.val[2]);

            if (hsv[0] != 100) {
                hsv[0] = 0;
                hsv[1] = 0;
                hsv[2] = 0;
            }
        }
    }

    cout << "$$ Hue(H), Saturation(S), Brightness(V) $$" << filenames[t] << " !! \n\n";
    cout << "H: " << h / corow * 360 / 180 << " % \n";  // 
    cout << "S: " << s / corow * 100 / 255 << " % \n";
    cout << "V: " << v / corow * 100 / 255 << " % \n\n";

    std::ofstream file("D:\\data\\dfdf\\result_4.txt", std::ios_base::app);
    file << v / corow * 100 / 255 << " \n"; // v value
    file.close();
}   

}

As you can see the imageAnalysis() function, I added std::string folder for the path of extracted images from video clip. However, when I applied this code, I have really weird results like below..

enter image description here

I thought I am supposed to get color value from every 24th image but as you see the results above, I got color values from all images in random order.

Thank you in advance.

It was really nice to learn how to code in efficient way!!

Answer 1

Just to clear the error you mentioned about CV_CAP_PROP_POS_MSEC in your comments:

when I apply CV_CAP_PROP_POS_MSEC to my code, I found some error messages like "CV_CAP_PROP_POS_MSEC is not defined."

A lot of the constant values are scoped in OpenCV. That means, CV_CAP_PROP_POS_MSEC is not defined, but cv::CV_CAP_PROP_POS_MSEC is. You can also obtain the FPS with cv::CAP_PROP_FPS .

Now to your code, I would actually do something that does not require to save and load the image, but rather pass the images to be processed, like this:

#include "opencv2/opencv.hpp"
#include <iostream>

int main(){
  cv::VideoCapture cap("D:\\data\\extra\\video200ul.avi"); 
  if (!cap.isOpened())  
  {
    std::cout << "Could not open video" << std::endl;
    return -1;
  }

  cv::Ptr<cv::BackgroundSubtractor> pMOG2 = cv::createBackgroundSubtractorMOG2(20, 16, true);

  cv::Mat fg_mask, frame;
  int count = 0;
  const int fps = 24; // you may set here the fps or get them from the video

  std::string name, folder;

  // with cap.read you can check already if the video ended
  while (cap.read(frame)) {
    // Background subtraction
    if (count % fps == 0) {
        pMOG2->apply(frame, fg_mask, 0.001);
        // Save foreground mask
        name = "mask" + std::to_string(count) + ".png";
        bool result = cv::imwrite("D:\\data\\extra\\" + name, frame);
        imageAnalysis(frame, count);
    }
    // at the end of the loop so that the first image is used
    ++count;
  }   
  cv::waitKey(0);
  return 0;
}

And the imageAnalysis function is defined as:

// You can pass cv::Mat as value, it is almost like a smart pointer
void imageAnalysis(cv::Mat frame, int count)
{
  cv::Mat ori_image = frame.clone();
  cv::rectangle(ori_image, Point(215, 98), Point(245, 110), Scalar(0, 255, 255), 1); 
  // each imshow needs a waitKey to update the window in which it is being shown
  cv::imshow("Original Image", ori_image);
  cv::waitKey(1);

  cv::Scalar sums;
  sums = cv::sum(ori_image);
  double totalSum = sums[0] + sums[1] + sums[2];

  std::ofstream output("D:\\data\\extra\\mask" + std::to_string(count) + ".txt");
  if (totalSum <= 0) 
  {
    std::cout << "$$ RGB percentage $$" << std::endl << std::endl;
    std::cout << "R: " << 100.0 / 3 << std::endl;
    std::cout << "G: " << 100.0 / 3 << std::endl;
    std::cout << "B: " << 100.0 / 3 << std::endl << std::endl;
    output << "$$ RGB percentage $$" << std::endl << std::endl;
    output << "R: " << 100.0 / 3 << std::endl;
    output << "G: " << 100.0 / 3 << std::endl;
    output << "B: " << 100.0 / 3 << std::endl << std::endl;
  }
else {
    std::cout << "$$ RGB percentage $$" << std::endl << std::endl;
    std::cout << "R: " << sums[2] / totalSum * 100 << std::endl; // red value
    std::cout << "G: " << sums[1] / totalSum * 100 << std::endl; // green value
    std::cout << "B: " << sums[0] / totalSum * 100 << std::endl << std::endl; // blue value
    output  << "$$ RGB percentage $$" << std::endl << std::endl;
    output  << "R: " << sums[2] / totalSum * 100 << std::endl; // red value
    output  << "G: " << sums[1] / totalSum * 100 << std::endl; // green value
    output  << "B: " << sums[0] / totalSum * 100 << std::endl << std::endl; // blue value
  }
}

Some comments of the code above, I replaced the cap >> frame to cap.read(frame) . It is the same functionality, but the later gives a bool result that is false if it could not grab the image, like if the video is over. I change the count add at the end, yo you get the frames 0,23,... this way the first one will be use as well. Finally, you should use the namespaces cv:: , std:: etc. This is just best practice, it avoids ambiguities and problems that may arise with certain libraries.

If you do not need the image in disk, but only the analysis, then remove the saving part and pass every frame to the imageAnalysis function, this way you may have more data for your statistics. Also, consider returning the cv:Scalar of sums in the function and then you can do some statistics of the whole second or the whole video.

If you have any question, feel free to ask in the comments.