How to detect whether two boxes are connected with each other or not?

Question

I am trying to detect box information, that is connected with others or not. And, if they're connected, to which one. Like in the image below:

ABC block is connected to DEF, DEF block is connected to JKL, DEF block is connected to MNO, and so on.

I am able to detect the boxes and the texts present inside the boxes.

Answer 1

I will start with the final output of my solution, just to be sure, that's the desired outcome:

GHI is connected with DEF.
MNO is connected with DEF.
MNO is connected with JKL.
DEF is connected with JKL.
DEF is connected with ABC.

I will outline the general idea, for details, please see the comments in the code:

1. Inverse binary threshold, to get white content on black background.
2. Find contours with hierarchy. We get the "inner" boxes when looking for the children of the one present, outer contour.
3. Get binary masks for each of those boxes. Within that masks, use pytesseract to get the text.
4. Dilate the masks to include the border.
5. Iterate all combinations of two boxes:

   • Copy the thresholded image. Remove all boxes not belonging to the current pair.

   • Crop that image w.r.t. to the two boxes of the current pair:

     

   • Find only outer contours: If there's exactly one contour, both boxes are connected.

That'd be full code:

import cv2
import itertools
import numpy as np
import pytesseract

# Read image
img = cv2.imread('hO0if.jpg', cv2.IMREAD_GRAYSCALE)

# Inverse binary threshold to get rid of JPG artifacts and inverse black/white
gray = cv2.threshold(img, 128, 255, cv2.THRESH_BINARY_INV)[1]

# Find contours and hierarchy w.r.t. the OpenCV version
cnts = cv2.findContours(gray, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
cnts, hier = cnts[0:] if len(cnts) == 2 else cnts[1:]

# Filter children of most outer contour; these are the "inner" boxes
boxes = [cnts[i] for i in np.argwhere(hier[..., 3].flatten() == 0).flatten()]

# Get masks of boxes
masks = [cv2.drawContours(np.zeros_like(img), [b], -1, 255, cv2.FILLED) for b in boxes]

# Get texts inside the boxes
rois = [cv2.boundingRect(cv2.bitwise_and(img, img, mask=m)) for m in masks]
texts = [pytesseract.image_to_string(img[r[1]:r[1]+r[3], r[0]:r[0]+r[2]]) for r in rois]
texts = [t.replace('\n', '').strip() for t in texts]

# Dilate masks
masks = [cv2.dilate(m, np.ones((11, 11))) for m in masks]

# Get all combinations of two boxes
combs = list(itertools.combinations(range(len(boxes)), 2))

# Iterate all combinations of two boxes
for c in combs:

    # Temporary image
    tmp = gray.copy()

    # Iterate all boxes not belonging to the current pair
    for i in (j for j in range(len(boxes)) if j not in c):

        # Remove those boxes
        r = cv2.boundingRect(cv2.bitwise_and(img, img, mask=masks[i]))
        tmp[r[1]:r[1]+r[3], r[0]:r[0]+r[2]] = 0

    # Crop image w.r.t. the boxes of the current pair
    r = cv2.boundingRect(cv2.bitwise_or(masks[c[0]], masks[c[1]]))
    tmp = tmp[r[1]:r[1]+r[3], r[0]:r[0]+r[2]]

    # Find outer contours w.r.t. the OpenCV version
    cnts = cv2.findContours(tmp, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    cnts = cnts[0] if len(cnts) == 2 else cnts[1]

    # If there's a single outer contour, both boxes are connected
    if len(cnts) == 1:
        print('{} is connected with {}.'.format(texts[c[0]], texts[c[1]]))

Right now, I'm not sure, whether checking for len(cnts) == 1 is sufficient. I could imagine, there might be examples, where the exclusion of the other boxes might lead to dead ends, which would be then counted as contours also. Maybe, in that case, additionally checking the contours' sizes would be needed.

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.16299-SP0
Python:        3.9.1
NumPy:         1.20.1
OpenCV:        4.5.1
pytesseract:   4.00.00alpha
----------------------------------------