OpenCV C++ Optical Flow based segmentation code throws Exception

Question

From one of the SO answers, for segmentation using optical flow Dense optical flow is required. I modified this code . It uses calcOpticalFlowFarneback function to calculate the dense flow field between two successive frames.

According to the SO answers here and here , I tried to compute the magnitude of displacement using

sqrt(displacement_in_x^2 + displacement_in_y^2)

the function returns displacement as CV_32FC2

This is my code :-

#include "stdafx.h"

#include "cv.h"
#include "highgui.h"
#include <iostream>


using namespace cv;
using namespace std;

void drawOptFlowMap(const Mat& flow, Mat& cflowmap, int step,
                    double scale, const Scalar& color)
{
    for(int y = 0; y < cflowmap.rows; y += step)
        for(int x = 0; x < cflowmap.cols; x += step)
        {
            const Point2f& fxy = flow.at<Point2f>(y, x);
            line(cflowmap, Point(x,y), Point(cvRound(x+fxy.x), cvRound(y+fxy.y)),
                 color);
            circle(cflowmap, Point(x,y), 2, color, -1);
        }
}

int main()
{
    VideoCapture cap("vip.avi");


    if( !cap.isOpened() )
        return -1;

    Mat prevgray, gray, flow, cflow, frame;
    double col = cap.get(CV_CAP_PROP_FRAME_WIDTH);
    double row = cap.get(CV_CAP_PROP_FRAME_HEIGHT);
    Mat flow_img = Mat::zeros(row,col,CV_8UC1);

    namedWindow("flow", 1);
   namedWindow("segmented",1);

   for(;;)
    {
        cap >> frame;
        cvtColor(frame, gray, CV_BGR2GRAY);

        if( prevgray.data )
        {
            calcOpticalFlowFarneback(prevgray, gray, flow, 0.5, 3, 15, 15, 5, 1.2, 0);
            cvtColor(prevgray, cflow, CV_GRAY2BGR);
            drawOptFlowMap(flow, cflow, 8, 1.5, CV_RGB(0, 255, 0));
            imshow("flow", cflow);
        }

        //This part I added to  compute Magnitude of flow
for(int i=0;i<gray.rows;i++)
    for(int j=0;j<gray.cols;j++)
        flow_img.at<uchar>(i,j)= cvRound(  sqrt ( pow (flow.at<Vec2f>(i,j)[0] ,2 )+ pow ( flow.at<Vec2f>(i,j)[1],2) ));

//threshold image
threshold(flow_img,flow_img,10,255,THRESH_BINARY); //some arbitrary thresh value 10

        imshow("segmented",flow_img);

        if(waitKey(30)>=0)
            break;

        std::swap(prevgray, gray);
    }
    return 0;
}

I am using Microsoft Visual C++ 2010 and there are no build errors. But the program wont run and throws the following :

In the console window I get :

I dont understand where I made the mistake. Can anyone please help me out. There were similar SO questions here and the answer is the code that I used. Still I get the problems mentioned above

you can find my input video here or here

Answer 1

Are you sure that the line`

const Point2f& fxy = flow.at<Point2f>(y, x);

is allowed? Point2f is not one of the types that is described in the documentation of the cv::Mat class. Try with cv::Vec2f as template parameter instead. You can then fill the Point2f variable with the result.

Next problem: in the line where you compute the 8-bit magnitude of the flow, it would be safer to use a saturate_cast operator:

flow_img.at<uchar>(i,j)= cv::saturate_cast<uchar>(cvRound(  sqrt ( pow (flow.at<Vec2f>(i,j)[0] ,2 )+ pow ( flow.at<Vec2f>(i,j)[1],2) )));

Not sure yet that it's causing your program to crash unfortunately.

What the error tells you is that you are calling an OpenCV function with wrong parameters. In debug mode go step-by-step in your program, so that we can know which one.

Answer 2

The code line

std::swap(prevgray, gray);

could cause problems, since the behaviour of std::swap is

c(a); a=b; b=c;

and in that way the =operator is used, that do not apply a deep copy of your input image. Better use

gray.copyTo(prevgray);

to allocate own memory for prevgray. Else the data pointer of prevgray points to the content of gray.