How do I find the rates TP, TN, FP, FN and measure the quality of segmented algorithm?

Question

Currently I am implementing a system that counts the number of passengers crossing a line of interest in a subway station. To measure the quality of the segmentation algorithm, I installed a video camera on the ceiling of the subway station and I recorded a video of 13 seconds.

The video of 13 seconds contains 412 frames.

Below is my questions:

1. To obtain the TPR (True Positive Rate) is necessary to analyze the frames manually, ie check every one of the 412 frames and count the times that were labeled correctly? While the FPR, is counted only the "false alarms"?

2. To test each algorithm, should I use the correction filters? Or just analyze them raw form (no filters)?

3. Detection of these rates cover when you're crossing the line or across your screen extension?

Here is a portion of the 412 frames

I ask your help to solve this problem.

Answer 1

You definitely have to manually tag each time a man is crossing the line. This part is crucial in order to be able to evaluate your algorithm correctly.

I suggest you create a groundtruth file which list all frame indices when someone cross the line. Your algorithm output should be of the same type - frame indices where it detects line crossing (if you are using segmentation or other technique it shouldn't affect the evaluation process).

The last thing you should do is to define a match criteria : if there is a real line crossing in frame_i and your algorithm detects a crossing in frame_j, then it will be count as true positive if abs(frame_i-frame_j) < threshold, otherwise it will be count as false positive.

Answer 2

The TP, TN, FP, FN in measure the quality of comparison between ground truth images and the output of your system. In your project you need to compare number of people who crossing a line and the computer by your program.