How to handle the mean Intersection Over Union (mIOU) for unknown class in semantic segmentation?

Question

I implemented a FCN network to do semantic segmentation. I am using Cityscapes as my dataset. As you know, there are some classes in Cityscapes that you ignore during the training and it is labeled as 255. I used weighted loss to ignore the loss for the unknown classes(set the loss to zero for unknown class). Now I want to exclude unknown class from my evaluation metric(mean Intersection Over Union (mIOU)).It is not clear for me how to exclude the unknown class at this point.

At the moment I am considering all the classes including the unknown class like this using tensorflow method:

 miou, confusion_mat = tf.metrics.mean_iou(labels=annotation, predictions=pred_annotation, num_classes=num_cls)

with tf.control_dependencies([tf.identity(confusion_mat)]):
    miou = tf.identity(miou)

I tried this , but it give an error for unbound label(for the unkonwn label)

miou, confusion_mat = tf.metrics.mean_iou(labels=annotation, predictions=pred_annotation, num_classes=(num_cls-1))

Answer 1

If you have a class that you want to ignore during the mIoU calculation, and you have access to the confusion matrix then you can do it like this:

1. ignore the miou calculated by tensorflow (since it considers all classes and that is not what you want)
2. remove row and column from the confusion matrix that correspond to the class you want to ignore
3. recalculate miou metric with the new confusion matrix

How to recalculate miou metric from the confusion matrix?

• iou for the first class: iou_0 = conf_mat[0,0] / (sum(conf_mat[0,:]) + sum(conf_mat[:,0]) - conf_mat[0,0])
• iou for the second class: iou_1 = conf_mat[1,1] / (sum(conf_mat[1,:]) + sum(conf_mat[:,1]) - conf_mat[1,1])
• ...
• in general for the class j : iou_j = conf_matrix[j,j] / (sum(conf_mat[j,:]) + sum(conf_mat[:,j]) - conf_mat[j,j])

At the end, sum and average all these per class iou to get miou .