寻找马虎手绘矩形的属性

Question

Image I'm working with:

I'm trying to find each of the boxes in this image. The results don't have to be 100% accurate, just as long as the boxes found are approximately correct in position/size. From playing with the example for square detection, I've managed to get contours, bounding boxes, corners and the centers of boxes.

There are a few issues I'm running into here:

1. bounding rectangles are detected for both the inside and the outside of the drawn lines.
2. some extraneous corners/centers are detected.
3. I'm not sure how to match corners/centers with the related contours/bounding boxes, especially when taking nested boxes into account.

Image resulting from code:

Here's the code I'm using to generate the image above:

import numpy as np
import cv2
from operator import itemgetter
from glob import glob
def angle_cos(p0, p1, p2):
    d1, d2 = (p0-p1).astype('float'), (p2-p1).astype('float')
    return abs( np.dot(d1, d2) / np.sqrt( np.dot(d1, d1)*np.dot(d2, d2) ) )
def makebin(gray):
    bin = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 5, 2)
    return cv2.bitwise_not(bin)
def find_squares(img):
    img = cv2.GaussianBlur(img, (11, 11), 0)
    squares = []
    points = []`
    for gray in cv2.split(img):
        bin = makebin(gray)
        contours, hierarchy = cv2.findContours(bin, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
        corners = cv2.goodFeaturesToTrack(gray,len(contours)*4,0.2,15)
        cv2.cornerSubPix(gray,corners,(6,6),(-1,-1),(cv2.TERM_CRITERIA_MAX_ITER | cv2.TERM_CRITERIA_EPS,10, 0.1))
        for cnt in contours:
            cnt_len = cv2.arcLength(cnt, True)
            if len(cnt) >= 4 and cv2.contourArea(cnt) > 200:
                rect = cv2.boundingRect(cnt)
                if rect not in squares:
                    squares.append(rect)
    return squares, corners, contours
if __name__ == '__main__':
    for fn in glob('../1 (Small).jpg'):
        img = cv2.imread(fn)
        squares, corners, contours = find_squares(img)
        for p in corners:
            cv2.circle(img, (p[0][0],p[0][3]), 3, (0,0,255),2)
        squares = sorted(squares,key=itemgetter(1,0,2,3))
        areas = []
        moments = []
        centers = []
        for s in squares:
            areas.append(s[2]*s[3])
            cv2.rectangle( img, (s[0],s[1]),(s[0]+s[2],s[1]+s[3]),(0,255,0),1)
        for c in contours:
            moments.append(cv2.moments(np.array(c)))
        for m in moments:
            centers.append((int(m["m10"] // m["m00"]), int(m["m01"] // m["m00"])))
        for cent in centers:
            print cent
            cv2.circle(img, (cent[0],cent[1]), 3, (0,255,0),2)
        cv2.imshow('squares', img)
        ch = 0xFF & cv2.waitKey()
        if ch == 27:
            break
    cv2.destroyAllWindows()             

Answer 1

I suggest a simpler approach as a starting point. For instance, morphological gradient can serve as a good local detector of strong edges, and threshold on it tends to be simple. Then, you can remove too small components, which is relatively easy for your problem too. In your example, each remaining connected component is a single box, so the problem is solved in this instance.

Here is what you would obtain with this simple procedure:

The red points represent the centroid of the component, so you could grow another box from there that is contained in the yellow one if the yellow ones are bad for you.

Here is the code for achieving that:

import sys
import numpy
from PIL import Image, ImageOps, ImageDraw
from scipy.ndimage import morphology, label

def boxes(orig):
    img = ImageOps.grayscale(orig)
    im = numpy.array(img)

    # Inner morphological gradient.
    im = morphology.grey_dilation(im, (3, 3)) - im

    # Binarize.
    mean, std = im.mean(), im.std()
    t = mean + std
    im[im < t] = 0
    im[im >= t] = 1

    # Connected components.
    lbl, numcc = label(im)
    # Size threshold.
    min_size = 200 # pixels
    box = []
    for i in range(1, numcc + 1):
        py, px = numpy.nonzero(lbl == i)
        if len(py) < min_size:
            im[lbl == i] = 0
            continue

        xmin, xmax, ymin, ymax = px.min(), px.max(), py.min(), py.max()
        # Four corners and centroid.
        box.append([
            [(xmin, ymin), (xmax, ymin), (xmax, ymax), (xmin, ymax)],
            (numpy.mean(px), numpy.mean(py))])

    return im.astype(numpy.uint8) * 255, box


orig = Image.open(sys.argv[1])
im, box = boxes(orig)

# Boxes found.
Image.fromarray(im).save(sys.argv[2])

# Draw perfect rectangles and the component centroid.
img = Image.fromarray(im)
visual = img.convert('RGB')
draw = ImageDraw.Draw(visual)
for b, centroid in box:
    draw.line(b + [b[0]], fill='yellow')
    cx, cy = centroid
    draw.ellipse((cx - 2, cy - 2, cx + 2, cy + 2), fill='red')
visual.save(sys.argv[3])

Answer 2

I see you have already got the answer. But I think there is a much more simpler,shorter and better method available in OpenCV to resolve this problem.

While finding contours, you are also finding the hierarchy of the contours. Hierarchy of the contours is the relation between different contours.

So the flag you used in your code, cv2.RETR_TREE provides all the hierarchical relationship.

cv2.RETR_LIST provides no hierarchy while cv2.RETR_EXTERNAL gives you only external contours.

The best one for you is cv2.RETR_CCOMP which provides you all the contour, and a two-level hierarchical relationship. ie outer contour is always parent and inner hole contour always is child.

Please read following article for more information on hierarchy : Contour - 5 : Hierarchy

So hierarchy of a contour is a 4 element array in which last element is the index pointer to its parent. If a contour has no parent, it is external contour and it has a value -1 . If it is a inner contour, it is a child and it will have some value which points to its parent. We are going to exploit this feature in your problem.

import cv2
import numpy as np

# Normal routines
img = cv2.imread('square.JPG')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(gray,50,255,1)

# Remove some small noise if any.
dilate = cv2.dilate(thresh,None)
erode = cv2.erode(dilate,None)

# Find contours with cv2.RETR_CCOMP
contours,hierarchy = cv2.findContours(erode,cv2.RETR_CCOMP,cv2.CHAIN_APPROX_SIMPLE)

for i,cnt in enumerate(contours):
    # Check if it is an external contour and its area is more than 100
    if hierarchy[0,i,3] == -1 and cv2.contourArea(cnt)>100:
        x,y,w,h = cv2.boundingRect(cnt)
        cv2.rectangle(img,(x,y),(x+w,y+h),(0,255,0),2)

        m = cv2.moments(cnt)
        cx,cy = m['m10']/m['m00'],m['m01']/m['m00']
        cv2.circle(img,(int(cx),int(cy)),3,255,-1)

cv2.imshow('img',img)
cv2.imwrite('sofsqure.png',img)
cv2.waitKey(0)
cv2.destroyAllWindows()

Result :

Answer 3

This question is related to python image recognition . A solution is given in squres.py demo .