How can I draw rectangle with MatOfKeyPoint for Text Detection | Java

Question

I am working on a real time text detection and recognition with OpenCV4Android. Recognition part is totally completed. However, I have to ask question about text detection. I' m using the MSER FeatureDetector for detection text.

This is the real time and calling the method part:

public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
    carrierMat = inputFrame.gray();
    carrierMat = General.MSER(carrierMat);
    return carrierMat;
}

And this is the basic MSER implementation:

private static FeatureDetector fd = FeatureDetector.create(FeatureDetector.MSER);
private static MatOfKeyPoint mokp = new MatOfKeyPoint();
private static Mat edges = new Mat();

public static Mat MSER(Mat mat) {
    //for mask
    Imgproc.Canny(mat, edges, 400, 450);
    fd.detect(mat, mokp, edges);
    //for drawing keypoints
    Features2d.drawKeypoints(mat, mokp, mat);
    return mat;
}

It works fine for finding text with edges mask.

I would like to draw a rectangles for clusters like this:

or this:

You can assume that I have the right points.

As you can see, fd.detect() method is returning a MatOfKeyPoint. Hence I' ve tried this method for drawing rectangle:

public static Mat MSER_(Mat mat) {
    fd.detect(mat, mokp);
    KeyPoint[] refKp = mokp.toArray();
    Point[] refPts = new Point[refKp.length];

    for (int i = 0; i < refKp.length; i++) {
        refPts[i] = refKp[i].pt;
    }
    MatOfPoint2f refMatPt = new MatOfPoint2f(refPts);
    MatOfPoint2f approxCurve = new MatOfPoint2f();

    //Processing on mMOP2f1 which is in type MatOfPoint2f
    double approxDistance = Imgproc.arcLength(refMatPt, true) * 0.02;
    Imgproc.approxPolyDP(refMatPt, approxCurve, approxDistance, true);

    //Convert back to MatOfPoint
    MatOfPoint points = new MatOfPoint(approxCurve.toArray());
    // Get bounding rect
    Rect rect = Imgproc.boundingRect(points);
    // draw enclosing rectangle (all same color, but you could use variable i to make them unique)
    Imgproc.rectangle(mat, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), Detect_Color_, 5);
    //Features2d.drawKeypoints(mat, mokp, mat);
    return mat;
}

But when I was trying to Imgproc.arcLength() method, it suddenly stopped. I gave a random approxDistance value for Imgproc.approxPolyDP() method like 0.1, it doesn' t work really efficiently.

So how can I draw rectangle for detected text?

Answer 1

I tested your code and had exactly the same problem. For now I still can't find the problem within. But I found a project using both "MSER" and "Morphological". you can find it here .

The project have very simple structure and the author put the text detection in "onCameraFrame" method just like you. I implemented the method from that project and it worked, but the result was still not very good.

If you seek better text detection tool, here's two of them.

1. Stroke Width Transform(SWT): A whole new method for finding text area. It's fast and efficient. however it is only available in c++ or python. you can find some example here .

2. Class-specific Extremal Regions using class ERFilter:An advanced version of the MSER. Unfortunately, it is only available in OpenCV 3.0.0-dev. You can't use it in current version of OpenCV4Android. The document is here .

To be honest I am new in this area(2 months), but I hope these information can help you finish your project.

(update:2015/9/13) I've translated a c++ method from a post . It works far better than the first github project I mentioned. Here is the code:

public void apply(Mat src, Mat dst) {
    if (dst != src) {
        src.copyTo(dst);
    }
    Mat img_gray,img_sobel, img_threshold, element;

    img_gray=new Mat();
    Imgproc.cvtColor(src, img_gray, Imgproc.COLOR_RGB2GRAY);

    img_sobel=new Mat();
    Imgproc.Sobel(img_gray, img_sobel, CvType.CV_8U, 1, 0, 3, 1, 0,Core.BORDER_DEFAULT);

    img_threshold=new Mat();
    Imgproc.threshold(img_sobel, img_threshold, 0, 255, Imgproc.THRESH_OTSU+Imgproc.THRESH_BINARY);

    element=new Mat();
    element = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(17, 3) );
    Imgproc.morphologyEx(img_threshold, img_threshold, Imgproc.MORPH_CLOSE, element);
    //Does the trick
    List<MatOfPoint>  contours=new ArrayList<MatOfPoint>();
    Mat hierarchy = new Mat();
    Imgproc.findContours(img_threshold, contours, hierarchy, 0, 1);
    List<MatOfPoint> contours_poly=new ArrayList<MatOfPoint>(contours.size());
    contours_poly.addAll(contours);

    MatOfPoint2f mMOP2f1,mMOP2f2;
    mMOP2f1=new MatOfPoint2f();
    mMOP2f2=new MatOfPoint2f();

    for( int i = 0; i < contours.size(); i++ )

        if (contours.get(i).toList().size()>100)
        { 
            contours.get(i).convertTo(mMOP2f1, CvType.CV_32FC2);
            Imgproc.approxPolyDP(mMOP2f1,mMOP2f2, 3, true );
            mMOP2f2.convertTo(contours_poly.get(i), CvType.CV_32S);
            Rect appRect=Imgproc.boundingRect(contours_poly.get(i));
            if (appRect.width>appRect.height) 
            {
                Imgproc.rectangle(dst, new Point(appRect.x,appRect.y) ,new Point(appRect.x+appRect.width,appRect.y+appRect.height), new Scalar(255,0,0));
            }
        }   

}