带有 Tensorflow 的多边形边界框

Question

I understand that the tensorflow API to train custom object detection datasets uses only rectangular bounding boxes, namely xmin, xmax, ymax, ymin. I also understand that a polygon bounding box will greatly improve detection accuracy as it removes any unnecessary information within the bounding box allowing for a far superior training dataset. I currently use labelImg to bound all my images for training and it does offer polygon boxes. My question is, is there a way to modify the code in the tensorflow API to work with polygon boxes as opposed to just rectangle boxes?

Answer 1

No, at this point you may be more interested at semantic segmentation like Mask R-CNN (not implemented in Tensorflow's object detection API).The models in the API have specific differentiable layers (thus trainable) that find bounding boxes. The degrees of freedom on a polygon model would be more complicated. Mask R-CNN somewhat solves the polygon problem by identifying the object, then segmenting what within the bounding box is actually the object vs background.

Here's some introduction to some of popular algorithms used in object detection and how they work:

https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4?gi=b386f4274020

Answer 2

Well, I modified my polygon. I simulated polygon inscribed in rectangle with the same dimensions of my image (filled with zeroes like [w=0,h=0,collor=0]) and used as mask for my training.

it's a snippet from where it occurs in my code:

im_w, im_h, color = img.shape
#
...
  image_masked, img_masks = mask_polygon_over_image(image, all_lists_poligon)
  #
  #convert to retangular over gray image
  tf_input_image = tf.image.resize(np.uint8(img), (im_w, im_h))
  tf_input_mask  = tf.image.resize(np.uint8(img_masks), (im_w, im_h))

I'm testing this now. The all_lists_poligon comes from an annotated JSON file

Just worked for me!

Take a look in the mask_polygon_over_image function:

   #
def mask_polygon_over_image(image, all_lists_poligon):
    new_image = Image.new(mode="RGB", size =(3000, 3000), color = (255, 255, 255))
    draw = ImageDraw.Draw(image,'RGBA')
    draw_new = ImageDraw.Draw(new_image)
    for list_polygon in all_lists_poligon:
        draw.polygon(list_polygon, fill=(50, 255, 50, 105),outline="yellow")
        draw_new.polygon(list_polygon, fill=(0, 185, 0),outline="yellow")
    return image, new_image