在具有嘈杂边的矩形周围找到已知大小的旋转边界框

Question

I'm trying to find a rotated bounding box around a less-than-perfect binarized image of a rectangle. The imperfections are always different: sometimes it's hollow, sometimes there's stuff inside, sometimes one of the edges is missing a chunk, sometimes there's an extra chunk somewhere on the edge, and they're always slightly rotated by a random amount, but the size and shape of the expected bounding box is always nearly the same absolute value in pixels.

Here's some samples of what I have as inputs (resized to fit better in the post):

And ideally I'd like to find a bounding box around the outside of the white rectangle (although I'm mostly just interested in the edges) like this:

(found by inverting one of the hollow ones, getting the largest connected component, and getting a rotatedrect of forced size)

So far I've tried just getting a rotatedrect and forcing a shape afterwards, which works for almost every case except for when there's an extra chunk along one of the edges. I've tried getting connected components to isolate parts of it and get bounding boxes around those, which works for every case as long as they're hollow. I've tried dilating and eroding the image, getting contours and hough lines to try to find only the four cornerpoints, but I've had no luck with that either. I've also looked online for anything useful to no avail.

Any help or ideas would be greatly appreciated.

Answer 1

My solution comprises two parts:

1. Find (upright) bounding box of the big white rectangle by finding the biggest connected component, fill all holes in it, find outside vertical and horizontal lines (Hough), get the bounding box by taking the min/max x/y coordinates.
2. Match a (filled) rectangle of given size with center at center of bounding box from step 1 at different angles, print the best match as result.

Following is a simple program demonstrating this approach. The arguments at the beginning (filename, size of know rectangle, angle search range) would normally be passed in from the command line.

    import cv2
    import numpy as np

    # arguments
    file = '1.png'
    w0, h0 = 425, 630  # size of known rectangle
    ang_range = 1      # posible range (+/-) of angle in degrees

    # read image
    img = cv2.imread(file, cv2.IMREAD_GRAYSCALE)
    h, w = img.shape

    # find biggest connceted components
    nb_components, output, stats, _ = cv2.connectedComponentsWithStats(img, connectivity=4)
    sizes = stats[:, -1]
    max_label, max_size = 1, sizes[1]
    for i in range(2, nb_components):
        if sizes[i] > max_size:
            max_label = i
            max_size = sizes[i]
    img2 = np.zeros(img.shape, np.uint8)
    img2[output == max_label] = 128

    # fill holes
    contours, _ = cv2.findContours(img2, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
    for contour in contours:
        cv2.drawContours(img2, [contour], 0, 128, -1)

    # find lines
    edges = cv2.Canny(img2, 50, 150, apertureSize = 3)
    lines = cv2.HoughLinesP(edges, 1, np.pi/180, 40)

    # find bounding lines
    xmax = ymax = 0
    xmin, ymin = w-1, h-1
    for i in range(lines.shape[0]):
        x1 = lines[i][0][0]
        y1 = lines[i][0][1]
        x2 = lines[i][0][2]
        y2 = lines[i][0][3]
        cv2.line(img2, (x1,y1), (x2,y2), 255, 2, cv2.LINE_AA)
        if abs(x1-x2) < abs(y1-y2):
            # vertical line
            xmin = min(xmin,x1,x2)
            xmax = max(xmax,x1,x2)
        else:
            # horizcontal line
            ymin = min(ymin,y1,y2)
            ymax = max(ymax,y1,y2)
    cv2.rectangle(img2, (xmin,ymin), (xmax,ymax), 255, 1, cv2.LINE_AA)
    cv2.imwrite(file.replace('.png', '_intermediate.png'), img2)

    # rectangle of known size centered at bounding box
    xc = (xmax + xmin) / 2
    yc = (ymax + ymin) / 2
    box = np.zeros(img.shape, np.uint8)
    box[int(yc-h0/2):int(yc+h0/2), int(xc-w0/2):int(xc+w0/2)] = 255

    # find best match of this rectangle at different angles
    smax = angmax = 0
    for ang in np.linspace(-ang_range, ang_range, 20):
       rm = cv2.getRotationMatrix2D((xc,yc), ang, 1)
       rotbox = cv2.warpAffine(box, rm, (w,h))
       s = cv2.countNonZero(cv2.bitwise_and(rotbox, img))
       if s > smax:
           smax = s
           angmax = ang

    # output and visualize result
    def draw_rotated_rect(img, size, center, angle, color, thickness):
        rm = cv2.getRotationMatrix2D(center, angle, 1)
        p0 = np.dot(rm,(xc-w0/2, yc-h0/2,1))
        p1 = np.dot(rm,(xc-w0/2, yc+h0/2,1))
        p2 = np.dot(rm,(xc+w0/2, yc+h0/2,1))
        p3 = np.dot(rm,(xc+w0/2, yc-h0/2,1))
        pnts = np.int32(np.vstack([p0,p1,p2,p3]) + 0.5).reshape(-1,4,2)
        cv2.polylines(img, pnts, True, color, thickness, cv2.LINE_AA)
        print(f'{file}: edges {pnts[0].tolist()}, angle = {angle:.2f}°')

    res = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
    draw_rotated_rect(res, (w0,h0), (xc,yc), angmax, (0,255,0), 2)
    cv2.imwrite(file.replace('.png', '_result.png'), res)

Intermediate results to show how it works (gray = filled biggest connected component, thick white lines = Hough lines, thin white rectangle = upright bounding box):
(to view the full size pictures click on them and then remove the final m before the file extension)

Visualization of results (green = rotated rectangle of known size):

Results (should eventually be clamped to [0,image size), -1 is due to floating point rotation):

1.png: edges [[17, -1], [17, 629], [442, 629], [442, -1]], angle = 0.00°
2.png: edges [[7, 18], [9, 648], [434, 646], [432, 16]], angle = 0.26°
3.png: edges [[38, 25], [36, 655], [461, 657], [463, 27]], angle = -0.26°
4.png: edges [[36, 14], [28, 644], [453, 650], [461, 20]], angle = -0.79°

As we see in image 3, the match is not perfect. This could be due to the example images that were shrinked to somewhat differing sizes and of course I didn't know the size of the known rectangle, so I just assumed an appropriate value for the demonstration.
If this occurs with real data too, you may want to not only vary the angle to find the best match, but also shift the matching box a couple of pixels up/down and right/left. See for instance section 8.1 of Dawson-Howe: A Practical Introduction to Computer Vision with OpenCV for further details.