Image segmentation based on optical flow

Question

I am using c++,OpenCV library and in my software, I have estimated the optical flow in a video. Now, I want to group some moving objects, eg moving cars. I have used a dense optical flow algorithm (Farneback).

My first thoughts so far are to use "k means" algorithm to do the clustering.

I have thought of using the results of the Farneback optical flow to compute the displacement of the frames in each direction as following :

Eg :

Let Dx be the displacement in x direction( either positive or negative) and Dy the displacement in y direction(either positive or negative) .

Then i pass the array [Dx,Dy] as an input to k means with k=2 clusters. I hope this will give a rough background / foreground substraction .

However i am facing problems in computing the displacemet because the output of calcOpticalFlowFarneback is InputOutputArray flow . Should I access this array using a function like that for example ? :

findDisplacements(const Mat& flow, int step) {
const Point2f& Dx,Dy;
const Point2f& fxy = flow.at<Point2f>(y, x);
Dx=Point(cvRound(x+fxy.x))-Point(x,y);
Dy=Point(cvRound(y+fxy.y))-Point(y,x);
}

Answer 1

A small example where the motion vectors are used as feature to cluster:

cv::Mat img0 = cv::imread("test0.png");    // Image at time t-1
cv::Mat img1 = cv::imread("test1.png");    // Image at time t
cv::Mat flow;
//estimate Optical Flow
cv::calcOpticalFlowFarneback(img0, img1, flow, 0.5, 3, 21, 20, 5, 1.1);
std::vector<cv::Point2f> samples(flow.rows * flow.cols);
// arange sample vector
int n = 0;
for( int r = 0; r < flow.rows; r++) {
  for( int c = 0; c < flow.cols; c++){
    samples[n++] = flow.at<cv::Point2f>(r,c);
}}
cv::kmeans(samples, ...

Answer 2

You can use the history of motion.

updateMotionHistory(silh, mhi, timestamp, MHI_DURATION);
calcMotionGradient(mhi, mask, orient, MAX_TIME_DELTA, MIN_TIME_DELTA, 3);
segmentMotion(mhi, segmask, regions, timestamp, MAX_TIME_DELTA);