how to detect all the rectangular boxes in the given image

Question

I tried to detect all the rectangles in image using threshold, canny edge and applied contour detection but it was not able to detect all the rectangles. Finally, I thought of detect the same using hough transforms but when I tried to detect lines in image, I'm getting all the lines. I need to detect only rectangular boxes in the image. Can someone help me? I'm new to opencv.

Input image

Code:

import cv2
import matplotlib.pyplot as plt
import numpy as np

img =  cv2.imread("demo-hand-written.png",-1)
#img = cv2.resize(img,(1280,720))
edges = cv2.Canny(img,180,200)
kernel = np.ones((2,2),np.uint8)
d = cv2.dilate(edges,kernel,iterations = 2)
e = cv2.erode(img,kernel,iterations = 2)  
#ret, th = cv2.threshold(img, 220, 255, cv2.THRESH_BINARY_INV)

lines = cv2.HoughLinesP(edges,1,np.pi/180,30, maxLineGap=20,minLineLength=30)
for line in lines:
    #print(line)
    x1,y1,x2,y2 = line[0]
    cv2.line(img,(x1,y1),(x2,y2),(0,255,0),3)
cv2.imshow("image",img)
cv2.waitKey(0)
cv2.destroyAllWindows()

Answer 1

You can use below code as a starting point.

img =  cv2.imread('demo-hand-written.png')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

thresh_inv = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)[1]

# Blur the image
blur = cv2.GaussianBlur(thresh_inv,(1,1),0)

thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)[1]

# find contours
contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]

mask = np.ones(img.shape[:2], dtype="uint8") * 255
for c in contours:
    # get the bounding rect
    x, y, w, h = cv2.boundingRect(c)
    if w*h>1000:
        cv2.rectangle(mask, (x, y), (x+w, y+h), (0, 0, 255), -1)

res_final = cv2.bitwise_and(img, img, mask=cv2.bitwise_not(mask))

cv2.imshow("boxes", mask)
cv2.imshow("final image", res_final)
cv2.waitKey(0)
cv2.destroyAllWindows()

Output:

Figure 1 : Detected rectangular boxes in the above image

Figure 2 : Detected rectangular contours in the original image

Answer 2

Suggestion:

Using Hough, detect the horizontal lines. Then take every line in turn and consider a window as large as the line and with a short height. Use Hough to detect the verticals in this window. This will give you candidate corners. (You can also try one window above the line and one below.)

Then by some additional local processing (?), confirm that the candidates are really box corners, and find the corner orientation. In the end, you should be able to join the corners in a way that makes sense geometrically?