How to detect rectangular items in image with Python

Question

I have found a plethora of questions regarding finding "things" in images using openCV, et al. in Python but so far I have been unable to piece them together for a reliable solution to my problem.

I am attempting to use computer vision to help count tiny surface mount electronics parts. The idea is for me to dump parts onto a solid color piece of paper, snap a picture, and have the software tell me how many items are in it.

The "things" differ from one picture to the next but will always be identical in any one image. I seem to be able to manually tune the parameters for things like hue/saturation for a particular part but it tends to require tweaking every time I change to a new part.

My current, semi-functioning code is posted below:

import imutils
import numpy
import cv2
import sys

def part_area(contours, round=10):
    """Finds the mode of the contour area.  The idea is that most of the parts in an image will be separated and that
    finding the most common area in the list of areas should provide a reasonable value to approximate by.  The areas
    are rounded to the nearest multiple of 200 to reduce the list of options."""
    # Start with a list of all of the areas for the provided contours.
    areas = [cv2.contourArea(contour) for contour in contours]
    # Determine a threshold for the minimum amount of area as 1% of the overall range.
    threshold = (max(areas) - min(areas)) / 100
    # Trim the list of areas down to only those that exceed the threshold.
    thresholded = [area for area in areas if area > threshold]
    # Round the areas to the nearest value set by the round argument.
    rounded = [int((area + (round / 2)) / round) * round for area in thresholded]
    # Remove any areas that rounded down to zero.
    cleaned = [area for area in rounded if area != 0]
    # Count the areas with the same values.
    counts = {}
    for area in cleaned:
        if area not in counts:
            counts[area] = 0
        counts[area] += 1
    # Reduce the areas down to only those that are in groups of three or more with the same area.
    above = []
    for area, count in counts.iteritems():
        if count > 2:
            for _ in range(count):
                above.append(area)
    # Take the mean of the areas as the average part size.
    average = sum(above) / len(above)
    return average

def find_hue_mode(hsv):
    """Given an HSV image as an input, compute the mode of the list of hue values to find the most common hue in the
    image.  This is used to determine the center for the background color filter."""
    pixels = {}
    for row in hsv:
        for pixel in row:
            hue = pixel[0]
            if hue not in pixels:
                pixels[hue] = 0
            pixels[hue] += 1
    counts = sorted(pixels.keys(), key=lambda key: pixels[key], reverse=True)
    return counts[0]


if __name__ == "__main__":
    # load the image and resize it to a smaller factor so that the shapes can be approximated better
    image = cv2.imread(sys.argv[1])

    # define range of blue color in HSV
    hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
    center = find_hue_mode(hsv)
    print 'Center Hue:', center

    lower = numpy.array([center - 10, 50, 50])
    upper = numpy.array([center + 10, 255, 255])
    # Threshold the HSV image to get only blue colors
    mask = cv2.inRange(hsv, lower, upper)
    inverted = cv2.bitwise_not(mask)

    blurred = cv2.GaussianBlur(inverted, (5, 5), 0)
    edged = cv2.Canny(blurred, 50, 100)
    dilated = cv2.dilate(edged, None, iterations=1)
    eroded = cv2.erode(dilated, None, iterations=1)

    # find contours in the thresholded image and initialize the shape detector
    contours = cv2.findContours(eroded.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    contours = contours[0] if imutils.is_cv2() else contours[1]

    # Compute the area for a single part to use when setting the threshold and calculating the number of parts within
    # a contour area.
    part_area = part_area(contours)
    # The threshold for a part's area - can't be too much smaller than the part itself.
    threshold = part_area * 0.5

    part_count = 0
    for contour in contours:
        if cv2.contourArea(contour) < threshold:
            continue

        # Sometimes parts are close enough together that they become one in the image.  To battle this, the total area
        # of the contour is divided by the area of a part (derived earlier).
        part_count += int((cv2.contourArea(contour) / part_area) + 0.1)  # this 0.1 "rounds up" slightly and was determined empirically

        # Draw an approximate contour around each detected part to give the user an idea of what the tool has computed.
        epsilon = 0.1 * cv2.arcLength(contour, True)
        approx = cv2.approxPolyDP(contour, epsilon, True)
        cv2.drawContours(image, [approx], -1, (0, 255, 0), 2)

    # Print the part count and show off the processed image.
    print 'Part Count:', part_count
    cv2.imshow("Image", image)
    cv2.waitKey(0)

Here's an example of the type of input image I am using: or this:

And I'm currently getting results like this:

The results clearly show that the script is having trouble identifying some parts and it's true Achilles heel seems to be when parts touch one another.

So my question/challenge is, what can I do to improve the reliability of this script?

The script is to be integrated into an existing Python tool so I am searching for a solution using Python. The solution does not need to be pure Python as I am willing to install whatever 3rd party libraries might be needed.

Answer 1

If the objects are all of similar types, you might have more success isolating a single example in the image and then using feature matching to detect them.

A full solution would be out of scope for Stack Overflow, but my suggestion for progress would be to first somehow find one or more "correct" examples using your current rectangle retrieval method. You could probably look for all your samples that are of the expected size, or that are accurate rectangles.

Once you have isolated a few positive examples, use some feature matching techniques to find the others. There is a lot of reading up you probably need to do on it but that is a potential solution.

A general summary is that you use your positive examples to find "features" of the object you want to detect. These "features" are generally things like corners or changes in gradient. OpenCV contains many methods you can use.

Once you have the features, there are several algorithms in OpenCV you can look at that will search the image for all matching features. You'll want one that is rotation invariant (can detect the same features arranged in different rotation), but you probably don't need scale invariance (can detect the same features at multiple scales).

My one concern with this method is that the items you are searching for in your images are quite small. It might be difficult to find good, consistent features to match on.

Answer 2

You're tackling a 2D object recognition problem, for which there are many possible approaches. You've gone about it using background/foreground segmentation, which is ok as you have control on the scene (laying down the background paper sheet). However this will always have fundamental limitations when the objects touch. A simple solution to your problem can be this:

1) You assume that touching objects are rare events (which is a fine assumption in your problem). Therefore you can compute the areas for each segmented region, and compute the median of these, which will give a robust estimate for the object's area. Let's call this robust estimate A (in squared pixels). This will be fine if fewer than 50% of regions correspond to touching objects.

2) You then proceed to measure the number of objects in each segmented region. Let Ai be the area of the ith region. You then compute the number of objects in each region by Ni=round(Ai/A). You then sum Ni to give you the total number of objects.

This approach will be fine as long as the following conditions are met: A) The touching objects do not significantly overlap B) You do not have objects lying on their sides. If you do you might be able to deal with this using two area estimates (side and flat). Better to eliminate this scenario if you can for simplicity. C) The objects are all roughly the same distance to the camera. If this is not the case then the areas of the objects (in pixels) cannot be modelled well by a single value. D) There are not partially visible objects at the borders of the image. E) You ensure that only the same type of object is visible in each image.