OpenCV 2.4.2 calcOpticalFlowPyrLK找不到任何积分

Question

I am using OpenCV 2.4.2 on Linux. I am writing in C++. I want to track simple objects (eg black rectangle on the white background). Firstly I am using goodFeaturesToTrack and then calcOpticalFlowPyrLK to find those points on another image. The problem is that calcOpticalFlowPyrLK doesn't find those points.

I have found code that does it in C, which does not work in my case: http://dasl.mem.drexel.edu/~noahKuntz/openCVTut9.html

I have converted it into C++:

int main(int, char**) {
    Mat imgAgray = imread("ImageA.png", CV_LOAD_IMAGE_GRAYSCALE);
    Mat imgBgray = imread("ImageB.png", CV_LOAD_IMAGE_GRAYSCALE);
    Mat imgC = imread("ImageC.png", CV_LOAD_IMAGE_UNCHANGED);

    vector<Point2f> cornersA;

    goodFeaturesToTrack(imgAgray, cornersA, 30, 0.01, 30);

    for (unsigned int i = 0; i < cornersA.size(); i++) {
        drawPixel(cornersA[i], &imgC, 2, blue);
    }

    // I have no idea what does it do
//    cornerSubPix(imgAgray, cornersA, Size(15, 15), Size(-1, -1),
//            TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 20, 0.03));

    vector<Point2f> cornersB;
    vector<uchar> status;
    vector<float> error;

    // winsize has to be 11 or 13, otherwise nothing is found
    int winsize = 11;
    int maxlvl = 5;

    calcOpticalFlowPyrLK(imgAgray, imgBgray, cornersA, cornersB, status, error,
            Size(winsize, winsize), maxlvl);

    for (unsigned int i = 0; i < cornersB.size(); i++) {
        if (status[i] == 0 || error[i] > 0) {
            drawPixel(cornersB[i], &imgC, 2, red);
            continue;
        }
        drawPixel(cornersB[i], &imgC, 2, green);
        line(imgC, cornersA[i], cornersB[i], Scalar(255, 0, 0));
    }

    namedWindow("window", 1);
    moveWindow("window", 50, 50);
    imshow("window", imgC);

    cvWaitKey(0);

    return 0;
}

ImageA: http://oi50.tinypic.com/14kv05v.jpg

ImageB: http://oi46.tinypic.com/4l3xom.jpg

ImageC: http://oi47.tinypic.com/35n3uox.jpg

I have found out that it works only for winsize = 11. I have tried using it on a moving rectangle to check how far it is from the origin. It hardly ever detects all four corners.

int main(int, char**) {
    std::cout << "Compiled at " << __TIME__ << std::endl;

    Scalar white = Scalar(255, 255, 255);
    Scalar black = Scalar(0, 0, 0);
    Scalar red = Scalar(0, 0, 255);
    Rect rect = Rect(50, 100, 100, 150);

    Mat org = Mat(Size(640, 480), CV_8UC1, white);
    rectangle(org, rect, black, -1, 0, 0);

    vector<Point2f> features;
    goodFeaturesToTrack(org, features, 30, 0.01, 30);
    std::cout << "POINTS FOUND:" << std::endl;
    for (unsigned int i = 0; i < features.size(); i++) {
        std::cout << "Point found: " << features[i].x;
        std::cout << " " << features[i].y << std::endl;
    }

    bool goRight = 1;

    while (1) {

        if (goRight) {
            rect.x += 30;
            rect.y += 30;
            if (rect.x >= 250) {
                goRight = 0;
            }
        } else {
            rect.x -= 30;
            rect.y -= 30;
            if (rect.x <= 50) {
                goRight = 1;
            }
        }

        Mat frame = Mat(Size(640, 480), CV_8UC1, white);
        rectangle(frame, rect, black, -1, 0, 0);

        vector<Point2f> found;
        vector<uchar> status;
        vector<float> error;
        calcOpticalFlowPyrLK(org, frame, features, found, status, error,
                    Size(11, 11), 5);

        Mat display;
        cvtColor(frame, display, CV_GRAY2BGR);

        for (unsigned int i = 0; i < found.size(); i++) {
            if (status[i]  == 0 || error[i] > 0) {
                continue;
            } else {
                line(display, features[i], found[i], red);
            }
        }

        namedWindow("window", 1);
        moveWindow("window", 50, 50);
        imshow("window", display);

        if (cvWaitKey(300) > 0) {
            break;
        }
    }

}

OpenCV implementation of Lucas-Kanade seems to be unable to track a rectangle on a binary image. Am I doing something wrong or does this function just not work?

Answer 1

The Lucas Kanade method estimates the motion of a region by using the gradients in that region. It is in a case a gradient descends methods. So if you don't have gradients in x AND y direction the method will fail. The second important note is that the Lucas Kanade equation

E = sum_{winsize} (Ix * u + Iy * v * It)²

is an first order taylor approximation of the intensity constancy constrain.

I(x,y,t) = I(x+u,y+v,t+1)

so an restriction of the method without level (image pyramids) is that the image needs to be a linear function. In practise this mean only small motions could be estimated, dependend from the winsize you choose. Thats why you use the levels, which linearise the images (It). So a level of 5 is a little bit to high 3 should be enough. The top level image has in your case a size of 640x480 / 2^5 = 20 x 15.

Finally the problem in your code is the line:

 if (status[i]  == 0 || error[i] > 0) {

the error you get back from the lucas kanade method is the resulting SSD that means:

error = sum(winSize) (I(x,y,0) - I(x+u,y+u,1)^2) / (winsize * winsize)

It is very unlikely that the error is 0. So finally you skip all features. I have good experiences by ignoring the error, that is just a confidence measure. There are very good alternative confidence measures as the Foreward/Backward confidence. You could also start experiments by ignoring the status flag if too much feaurtes are discard

Answer 2

KLT does point tracking by finding a transformation between two sets of points regarding a certain window. The window size is an area over which each point will be chased in order to match it on the other frame.

It is another algorithm based on gradient that find the good features to track.

Normally KLT uses a pyramidal approach in order to maintain tracking even with big movements. It probably uses at "maxLevel" times for the "window sized" you specified.

Never tried KLT on binary images. The problem might be on KLT implementation that begin the search in a wrong direction and then just lost the points. When you change the windows size then the search algorithm changes also. On you're picture you have only 4 interest point maximum and only on 1 pixel.

These are parameters you're interested in :

winSize – Size of the search window at each pyramid level
maxLevel – 0-based maximal pyramid level number. If 0, pyramids are not used (single level), if 1, two levels are used etc.
criteria – Specifies the termination criteria of the iterative search algorithm (after the specified maximum number of iterations criteria.maxCount or when the search window moves by less than criteria.epsilon

Suggestion :

• Did you try with natural pictures ? (two photos for instance), you'll have much more features to track. 4 or less is quite hard to keep. I would try this first