get bounding boxes with maximum confidence pandas opencv python

Question

I have a Symbol detection algorithm, which can be output from template matching/ faster rcnn or combining the results from both of them, which gives me the coordinates filename,xmin, ymin, xmax, ymax, class, confidence .

The issue is that there are multiple bounding boxes occurring for the same object. How I can efficiently filter these boxes and get only the boxes with maximum confidence in the area removing duplicates.

sample Image:

The sample coordinate data frame:

df=pd.DataFrame({'filename':['dummyfile.jpg']*5, class=['cube']*5, xmin':[88,87,65,492,470],'ymin':[87,111,110,187,184],'xmax':[197,198,174,603,578],
              'ymax':[198,220,221,295,295],'confidence':[0.99,0.88,0.95,0.89,0.83]})

  class  confidence       filename  xmax  xmin  ymax  ymin
0  cube        0.99  dummyfile.jpg   197    88   198    87
1  cube        0.88  dummyfile.jpg   198    87   220   111
2  cube        0.95  dummyfile.jpg   174    65   221   110
3  cube        0.89  dummyfile.jpg   603   492   295   187
4  cube        0.83  dummyfile.jpg   578   470   295   184

Image representation:

Expected output:

I tried filtering using confidence as a threshold, but it will affect the recall of the solution. How to remove these duplicates making use of IoU?

Answer 1

This is the solution which I came up with.

Creating a unique key for each bounding box

df['key']=df['xmin'].astype(str)+'_'+df['ymin'].astype(str)+'_'+df['xmax'].astype(str)+'_'+df['ymax'].astype(str)

Making an outer join of all the rows based on filename. (for calculating IoU)

###copy df
df_1=df.copy()

###Renaming df columns with _1 suffix
df_cols=df.columns.tolist()
df_cols.remove('filename')
new_cols=[col+'_1' for col in df_cols]
new_col_dict=dict(zip(df_cols,new_cols))
df_1.rename(columns=new_col_dict,inplace=True)

### Outer joining both dataframes
newdf=pd.merge(df,df_1,'outer',on='filename')

Outer join sample:

Function to find IoU of each row

def IOU(df):
    '''funtion to calulcate IOU within rows of dataframe'''
    # determining the minimum and maximum -coordinates of the intersection rectangle
    xmin_inter = max(df.xmin, df.xmin_1)
    ymin_inter = max(df.ymin, df.ymin_1)
    xmax_inter = min(df.xmax, df.xmax_1)
    ymax_inter = min(df.ymax, df.ymax_1)

    # calculate area of intersection rectangle
    inter_area = max(0, xmax_inter - xmin_inter + 1) * max(0, ymax_inter - ymin_inter + 1)

    # calculate area of actual and predicted boxes
    actual_area = (df.xmax - df.xmin + 1) * (df.ymax - df.ymin + 1)
    pred_area = (df.xmax_1 - df.xmin_1 + 1) * (df.ymax_1 - df.ymin_1+ 1)

    # computing intersection over union
    iou = inter_area / float(actual_area + pred_area - inter_area)

    # return the intersection over union value
    return iou

Calculating IoU for each row and filtering rows with ioU<0.4

newdf['iou']= newdf.apply(IOU, axis = 1)
### filtering all iou<0.4
newdf=newdf[newdf['iou']>=0.4]

Getting the best match for each key

once we have the IoU match dataframe, parse through each unique key bounding box, get the value with max confidence for each unique key with iou>0.4

best_df=pd.DataFrame()
for i, v in df.iterrows():
    key=v['key']
    iou_match=newdf[newdf['key']==key]
    iou_match.sort_values('confidence_1',ascending=False,inplace=True)
    iou_match=iou_match.reset_index()
    best_match=iou_match.loc[0,['filename','class_1','xmin_1','ymin_1','xmax_1','ymax_1','confidence_1']]
    best_df=best_df.append(best_match,ignore_index=True)

best_df now looks like:

  class_1  confidence_1       filename  xmax_1  xmin_1  ymax_1  ymin_1
0  cube          0.99  dummyfile.jpg   197.0    88.0   198.0    87.0
1  cube          0.99  dummyfile.jpg   197.0    88.0   198.0    87.0
2  cube          0.99  dummyfile.jpg   197.0    88.0   198.0    87.0
3  cube          0.89  dummyfile.jpg   603.0   492.0   295.0   187.0
4  cube          0.89  dummyfile.jpg   603.0   492.0   295.0   187.0

Removing duplicates to get unique best matches

best_df.drop_duplicates(inplace=True)

Final result:

  class_1  confidence_1       filename  xmax_1  xmin_1  ymax_1  ymin_1
0  cube          0.99  dummyfile.jpg   197.0    88.0   198.0    87.0
3  cube          0.89  dummyfile.jpg   603.0   492.0   295.0   187.0

Answer 2

Instead of the iteration and drop_duplicates, it should be easier to do a filter followed by groupby and max aggregation to pick the key with the highest confidence (while making sure its permutation row gets filtered out).

best_df = df.apply(lambda g: g['confidence']>g['confidence_1']).groupby('key').filter(agg({'confidence_1':'max'})