Detect Image object from UIImage using OpenCV

Question

I am completely new for OpenCV, but during googling I came to know about Object Detection & Edge Detection. But, Still not able to figure out proper way to Detect Image from ScreenShot.

For Example, If I pass an image having photo inside it like below, then I need to extract that Photo from source Image.

EDIT After following Answer of @Amitay Nachmani, I tried to implement the following code up to step 4.

-(UIImage*)processImage:(UIImage*)sourceImage{

    cv::Mat processMat;
    UIImageToMat(sourceImage, processMat);

    cv::Mat grayImage;
    cvtColor(processMat, grayImage, CV_BGR2GRAY);

    cv::Mat cannyImage;
    cv::Canny(grayImage, cannyImage, 0, 50);


    cv::Vec2f lines2;
    std::vector<cv::Vec2f> lines;
    cv::HoughLines(cannyImage, lines, 1, CV_PI/180, 300);

    size_t sizeOfLine = lines.size();
    for(size_t i=0;i<sizeOfLine;i++){
        float rho = lines[i][0], theta = lines[i][1];

        if(rho==0){
            cv::Point pt1,pt2;
            double a = cos(theta), b = sin(theta);
            double x0 = a*rho, y0 = b*rho;
            pt1.x = cvRound(x0 + 1000*(-b));
            pt1.y = cvRound(y0 + 1000*(a));
            pt2.x = cvRound(x0 - 1000*(-b));
            pt2.y = cvRound(y0 - 1000*(a));

            cv::line(cannyImage, pt1, pt2, cv::Scalar(255,0,0),2.0);
        }
    }

    UIImage *result = MatToUIImage(cannyImage);
    return result;
}

From above code, I got generated following Image.

EDIT 2 I revised code by replacing Condition if(rho==0) with if(theta==0)

This resulted in below Image

But, Still What to do next ? I am bit confused in next Steps.

Answer 1

I am not completely sure but, did you try template matching technique? If you are using c++ to code opencv: http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html

I hope this will be helpful to find cross-correlation between template (your source image) and your test image (screenshot).

In the link below you will find a complete example of how to apply and draw template matching.

Hope this helps.

Cheers.

Unai.

Answer 2

If you know that the image is always between the second horizontal line and the third i would do the following:

1. Convert to gray scale (opencv cvtColor)
2. Run Canny edge detection (opecv Canny())
3. Find lines using Hough lines (opencv HoughLines() http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_imgproc/py_houghlines/py_houghlines.html )
4. Take only the 4 most prominent horizontal lines ( to take the horizontal ones you need theta = 90)
5. Sort the lines you find according to the y coordinate
6. Crop the image between the second and third line

Answer 3

I completely agree with the post below, this is the best solution but unfortunately, I guess that @Mrug development will be targeted to smartphone devices, and canny edge detection and hough line transform are computationally very expensive from those platforms.

May be you can use Sobel derivates which are designed to calculate horizontal and vertical derivates.

These links may help you:

Sobel Derivates http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/sobel_derivatives/sobel_derivatives.html

Canny edge detectors http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/canny_detector/canny_detector.html

Hough transform:

http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html