How to Compute the Structure Tensor of an Image using OpenCV

Question

I am trying to implement an application that uses images to search for similar images in a large image database. I am developing an image descriptor to use for this search and I would like to combine color information with some gradient information. I have seen structure tensors used in this domain to find the main gradient direction in images or sub-images.

I would like to take an image, divide it into grid of sub-images, for example, 4x4 grid (in total 16 sub-images) and then find the leading gradient direction of each cell. To find the leading gradient direction I want to see if computing the structure tensor for each cell can give good representation of the image gradient and lead to improved image matching. Is this a good idea or a bad idea? The idea was to get a feature vector similar to the idea in section 3.2 in this paper http://cybertron.cg.tu-berlin.de/eitz/pdf/2009_sbim.pdf

Dividing the image into sub-images(cells) is trivial and with opencv I can compute the partial derivatives using the Sobel function.

Mat dx, dy; 
Sobel(im, dx, CV_32F, 1, 0, 3, 1, 0, BORDER_DEFAULT);
Sobel(im, dy, CV_32F, 0, 1, 3, 1, 0, BORDER_DEFAULT); 

Computing dx^2, dy^2 and dxy should not be a problem, but I am not sure how I can compute the structure tensor matrix and use the tensor matrix to find the main gradient direction for an image or sub-image. How can I implement this with OpenCV?

EDIT Okay, this is what I have done.

    Mat _im; // Image to compute main gradient direction for. 
    cvtColor(im, _im, CV_BGR2GRAY);
    GaussianBlur(_im, _im, Size(3, 3), 0, 0, BORDER_DEFAULT); //Blur the image to remove unnecessary details. 
    GaussianBlur(_im, _im, Size(5, 5), 0, 0, BORDER_DEFAULT);
    GaussianBlur(_im, _im, Size(7, 7), 0, 0, BORDER_DEFAULT);

    // Calculate image derivatives 
    Mat dx2, dy2, dxy; 
    Sobel(_im, dx2, CV_32F, 2, 0, 3, 1, 0, BORDER_DEFAULT); 
    Sobel(_im, dy2, CV_32F, 0, 2, 3, 1, 0, BORDER_DEFAULT);
    Sobel(_im, dxy, CV_32F, 1, 1, 3, 1, 0, BORDER_DEFAULT);

    Mat t(2, 2, CV_32F); // tensor matrix

    // Insert values to the tensor matrix.
    t.at<float>(0, 0) = sum(dx2)[0];
    t.at<float>(0, 1) = sum(dxy)[0];
    t.at<float>(1, 0) = sum(dxy)[0];
    t.at<float>(1, 1) = sum(dy2)[0];

    // eigen decomposition to get the main gradient direction. 
    Mat eigVal, eigVec;
    eigen(t, eigVal, eigVec);

    // This should compute the angle of the gradient direction based on the first eigenvector. 
    float* eVec1 = eigVec.ptr<float>(0);
    float* eVec2 = eigVec.ptr<float>(1);
    cout << fastAtan2(eVec1[0], eVec1[1]) << endl;
    cout << fastAtan2(eVec2[0], eVec2[1]) << endl;

Is this approach correct?

---

Using this image the application outputs 44.9905, 135.01. This gives 0, 90.

---

When I use a part of a real image I get 342.743, 72.7425, which I find odd. I expected to get an angle along the color change (90ish).

After testing I am not sure if my implementation is correct, so any feedback or comments on this are welcomed.

Answer 1

I believe your problem is that you are computing second order derivatives instead of squaring first order derivatives. It should be something like this instead:

// Calculate image derivatives 
Mat dx, dy; 
Mat dx2, dy2, dxy;
Sobel(_im, dx, CV_32F, 1, 0); 
Sobel(_im, dy, CV_32F, 0, 1);
multiply(dx, dx, dx2);
multiply(dy, dy, dy2);
multiply(dx, dy, dxy);

PS Oh, by the way, there is no need to do Gaussian blurring over and over again. Just use a bigger kernel and blur once. DS