C++ calculate 2 3x3 Covariance matrices for each region in image

Question

I have an image img in c++ of size mxn of type CV_8UC3 in OpenCV. I have another vector b of size 1xn splitting img "horizontally" into two parts:

upper_part = {(row,col)|1<=col<=n, 1<=row<=b(1,col)}

lower_part = {(row,col)|1<=col<=n, b(1,col)<row<=m} ,

where 1<=b(1,col)<=m (1<=col<=n).

For these two parts of the image I'd like to have the covariance matrices M_u and M_l sort of "per" channel. This means the resulting matrices should have size 3x3 and should be derived like:

M_u = 1/(N_u-1) * sum_{(row,col)\\in upper_part} (img(row,col)-mu_u)*(img(row,col)-mu_u)^T ,

where N_u is the number of elements in the upper part, mu_u a 3x1 vector describing the average RGB values in the upper part and img(row,col) is a 3x1 vector with the RGB values of img at position (row,col) . M_l is calculated equivalently with N_l and mu_l considering the lower_part .

Furthermore, I also (sometimes) have to calculate the covariance for an CV_8UC1 image. Of course then the matrix is just a scalar.

Is there a solution primarily for the CV_8UC3 type and if yes is there a solution which also works for the CV_8UC1 type image?

My current solution is to iterate over each pixel and calculate it by getting the values with img.at<Vec3b>(row,col) or img.at<unsigned char>(row,col) respectively (first for the mean, then for the covariance, thus two loops over all pixels), but what I've heard and now see is that this function is quite inefficient/slow. As I've to do this process of calculating M_u and M_l within a loop I'd like to derive the covariances efficiently. Any ideas?

Thank you.

PS: m~1280 and n~960 .

Answer 1

Calculation of covariance within a single loop of iteration of all pixels is possible.

I have the following code that iterates through the entire set of pixels of an image,just one time and calculate the co-variance matrix. This can be very well extended to your case of split images.

   {
    //img is a CV_8UC3 image in RGB format
    Vec3f sumOfPixels=Vec3f(0,0,0);
    float sumRR=0, sumRG=0, sumRB=0, sumGG=0, sumGB=0, sumBB=0;
    Mat covarianceMat = Mat::zeros(3, 3, CV_32FC1);
    for(int r= 0; r < img.rows; ++r) {
        for(int c=0; c < img.cols; ++c) {
            const Vec3b &currentPixel = img.at<Vec3b>(Point(c,r));
            sumOfPixels += Vec3b(currentPixel[0], currentPixel[1], currentPixel[2]);
            sumRR += currentPixel[0] * currentPixel[0];
            sumRG += currentPixel[0] * currentPixel[1];
            sumRB += currentPixel[0] * currentPixel[2];
            sumGG += currentPixel[1] * currentPixel[1];
            sumGB += currentPixel[1] * currentPixel[2];
            sumBB += currentPixel[2] * currentPixel[2];
        }
    }
    int nPixels = img.rows * img.cols;
    assert(nPixels > 0);
    Vec3f avgOfPixels = sumOfPixels / nPixels;
    covarianceMat.at<float>(0,0) = sumRR/nPixels - avgOfPixels[0]*avgOfPixels[0];
    covarianceMat.at<float>(0,1) = sumRG/nPixels - avgOfPixels[0]*avgOfPixels[1];
    covarianceMat.at<float>(0,2) = sumRB/nPixels - avgOfPixels[0]*avgOfPixels[2];

    covarianceMat.at<float>(1,1) = sumGG/nPixels - avgOfPixels[1]*avgOfPixels[1];
    covarianceMat.at<float>(1,2) = sumGB/nPixels - avgOfPixels[1]*avgOfPixels[2];
    covarianceMat.at<float>(2,2) = sumBB/nPixels - avgOfPixels[2]*avgOfPixels[2];

    covarianceMat.at<float>(1,0) =  covarianceMat.at<float>(0,1);
    covarianceMat.at<float>(2,0) =  covarianceMat.at<float>(0,2);
    covarianceMat.at<float>(2,1) =  covarianceMat.at<float>(1,2);
    cout << "covariance of image: " << covarianceMat << endl;

}

In the case of the calculating the covariance for full image (ie: not split image), you could check whether the covariance is correct by using opencv's 'calcCovarMatrix' as well.

    Mat img_copy = img;
    assert(img.type() == img_copy.type());
    img_copy = img.reshape(1, img.rows *img.cols);
    cv::Mat covar, mean;
    cv::calcCovarMatrix(img_copy, covar, mean, CV_COVAR_NORMAL | CV_COVAR_ROWS );
    covar /= (img.rows * img.cols);
    std::cout << "covariance through opencv: " << covar << std::endl;