sklearn如何计算两个二进制输入的roc曲线下的面积？

Question

I noticed that sklearn has the following function:

sklearn.metrics.roc_auc_score()

which takes as input ground_truth and prediction.

For example,

ground_truth = [1,1,0,0,0]
prediction = [1,1,0,0,0]

sklearn.metrics.roc_auc_score(ground_truth, prediction) returns 1

My problem is that I can't figure out how sklearn calculates the area under the ROC curve with two binary inputs. Isn't the ROC curve derived by moving the class assignment threshold, and calculating the false alarm and hit rate for each threshold? With two binary inputs, shouldn't you only have one (false alarm, hit rate) measurement?

Many thanks!

Answer 1

You're correct that with binary predictions you'll only have a single threshold/measurement for the curve. I didn't understand it myself so I ran the code with a ton of print statements both for the sklearn tutorial and then with a purely binary example. All the magic is happening in sklearn.metrics._binary_clf_curve

The "thresholds" are distinct prediction scores. For any binary classifier that outputs purely ones and zeros you're going to get two thresholds - 1 and 0 (they're sorted internally from highest to lowest). At the 1 threshold, a prediction score of >=1 is true and anything below that (only 0 in this case) is considered false, and the TP and FP rates are calculated from that. In all cases, the last threshold categorizes everything as true so the TP and FP rates will both be 1.

It appears then that to generate a correct ROC curve for a sklearn classifier you'd use clf.predict_proba() rather than predict() . Or, maybe predict_log_proba() ? I'm not sure if it would make any difference