OpenCV and iOS, extracting/cropping image into matching parts

Question

I've been struggling trying to find a solution to this problem, and it may be simpler than I'm thinking.

I'm currently building an app that needs to analyze an image using TensorFlow. That part I've been able to get up and working no problem (think the "not a hotdog" app).

The problem I'm running into is that the image I need to analyze is going to contain several of the individual images I'm after.

TLDR ; The best way I can think to explain, is that I have trained a model to analyze an individual egg. Now I take a picture of an egg carton (somewhat symmetrical, and orderly, but could be different angles/lighting/etc.) How would I got about extracting the individual eggs from the picture to analyze individually? The other key piece is that I need them ordered (top to bottom, or bottom to top, NOT by best match if, for example, using the OpenCV matchTemplate method)

I would love to share code, but so far it's all been research, which is what led me to OpenCV, and so far I've only added it as a library dependancy. I've looked at template matching, flood filling, etc. I just can't tie it all together to be able to "cut up" the image into it's individual pieces.

I hope this is enough to go off of, but please let me know and I can try to add more detail.

As a reference I've read through these examples:

https://samwize.com/2013/06/09/using-opencv-to-match-template-multiple-times/

https://www.toptal.com/machine-learning/real-time-object-detection-using-mser-in-ios ...and numerous others, these ones just seemed the most relevant to what I'm trying to accomplish.

EDIT As an example, I found this image that provides a good comparison to what I'll be working with. Given this image of lego blocks, how would I export the individual pieces? I'm not too concerned with extra data (ie doesn't need to crop it exactly, just need to separate the pieces). The other key thing is that I need them ordered from top to bottom (in the example image the blue block would be #1, red #2, etc.)

Answer 1

Not having a sample image to test, I can just give you a theoretical way on how to proceed.

You can just use OTSU binarization to the original image, and get a black and white image, where each egg is a white blob on a black background.

(optional: run some morphological operation like erosion, opening, ecc to separate touching eggs)

On this image, you can use cv::findContours to extract the borders of every single egg.

Now on every border, you can use cv::minAreaRect to find the bounding box of every egg.

Filter out, using some thresholding on the area of the bounding box, the false positives.

Now you got an array of rotated bounding boxes that you can use for extracting the same regions in the original image and feed them as input to your trained network (and since you're using tensorflow, you can create a batch of images and feed all the batch as input to your network, to run the evaluation in parallel)

Here's a skech in python ("image.png" is your lego blocks image):

import cv2
import numpy as np

def main():
    img = cv2.imread("image.png", 0)
    threshold, binary = cv2.threshold(img,0,255,cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)

    cv2.imshow("bin",binary)

    im2, contours, hierarchy = cv2.findContours(binary, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    draw = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
    #cv2.drawContours(img, contours, 0, (255,0, 0), 2)

    boxes = []
    for contour in contours:
        #cv2.drawContours(img, [contour], 0, (0,255,0), 2)
        area = cv2.contourArea(contour)
        # heuristic: ignore small objects and noise
        if area < 50:
            continue
        box = cv2.boundingRect(contour)
        boxes.append(box)


    feed_to_tensorflow = []
    for box in boxes:
        #extract roi
        x, y, width, height = box
        roi = img[y:y+height, x:x+width]
        feed_to_tensorflow.append(roi)
        cv2.imshow("box", feed_to_tensorflow[-1])
        cv2.waitKey()


    cv2.imshow("img", img)
    cv2.waitKey()


if __name__ == "__main__":
    main()