二元分类问题中每个概率临界值的准确度（python sklearn 准确度）

Question

Imagine a binary classification problem. Let's say I have 800,000 predicted probabilities stored in pred_test . I define a cutoff as any value in pred_test such that the values that are greater than or equal to cutoff are assigned the value 1 and the values that are smaller than cutoff are assigned the value 0.

Is there a function in sklearn that returns the accuracy of the model for each cutoff in pred_train ? I would like to see the accuracy of the model as a function of each cutoff to systematically pick a cutoff.

I tried the following:

_list = []
for cutoff in np.unique(np.sort(pred_test)):
    binary_prediction = np.where(pred_test >= cutoff, 1, 0)
    _list.append( (cutoff, binary_prediction == y_test).sum() / len(pred_test) )

Here, y_test is the ground truth (an array with the observed outcomes for each of the 800,000 rows). This code returns a list where each value contains the cutoff and its corresponding accuracy score.

The object pred_test has around 600,000 different values, so I am iterating 600,000 or so times. The above code is working, but it's taking a very long time to finish. Is there a more efficient way to do this? My bet is that sklearn already has a function that does this.

Answer 1

here is some similiar thread to check it: Getting the maximum accuracy for a binary probabilistic classifier in scikit-learn

There is no built-in function for that in scikit-learn. I think the reason why this is not implemented is that you will have the chance to overfit, you basically will tune your train set to a baseline that is risky for the test set.