Accuracy Score for a vector of predictions using Logistic Regression in Python

Question

I'm doing the multiclass classification using Logistic Regression approach. Basically I know that if I use accuracy_score () function (for example, from sklearn library) it will calculate me the accuracy of distinct value to distinct value like this:

y_pred = [0, 2, 1, 3]
y_true = [0, 1, 2, 3]
accuracy_score(y_true, y_pred)
0.5

But I want to get the accuracy_score_new () function for a vector of top Logistic Regression predictions for each label (from predict_proba) and calculates whether the true label is in this interval like this:

y_pred = [[0,1,3] [2,1,4], [1,2,5] [3,7,9]]
y_true = [0, 1, 2, 3]
accuracy_score_new(y_true, y_pred)
1

The accuracy_score_new in this example will be equal to 1 because the classifier predicts that the label is in the interval. How can this function be done?

Answer 1

Yes you can do that using the make_scorer function in sklearn. The idea is that you define your custom function assuming it gets the two parameters y_true and y_pred. You can also add any additional parameters if you want.

Here is an example : Custom scoring function

Here is another example : Using MSE and R2 score at the same time

This answer might be of some help too.

Answer 2

Accuracy is just (matching values /total values).

So in your case it will be something like:

def accuracy_score_new(y_pred, y_true):
    matched = 0
    for y_p, y_t in zip(y_pred, y_true):
        if y_t in y_p:
            matched = matched + 1

    return (matched / (float) len(y_true))