Sklearn.metrics.classification_report 混淆矩阵问题？

Question

Firstly, thank you for reading my question - I hope this is the right place for this.

I am coding up sensitivity, specificity and precision calculations from a confusion matrix from scratch. I have the following confusion matrix for 4 classes.

                    True Class
                   1   2   3   4

           1   [[  0   1   3   0]
Predicted  2    [  0 181  23   0]
Class      3    [  0  17  53  14]
           4    [  0   3  22  77]]

When I use Sklearn.metrics.classification_report this is what I get:

precision  recall  f1-score   support

  0.00      0.00      0.00         4
  0.89      0.89      0.89       204
  0.52      0.63      0.57        84
  0.85      0.75      0.80       102

However, for precision and recall I get (ie the values for precision and recall are flippped):

precision  recall
  0.0       nan
  0.887     0.896
  0.631     0.524
  0.755     0.846

For each class I calculate the following true positives, false positives, true negatives and false negatives:

class Tp  Fp  Tn   Fn
1     0   4   390  0
2     181 23  169  21
3     53  31  262  48
4     77  25  278  14

The formulas that I'm using ( https://en.wikipedia.org/wiki/Confusion_matrix ) are:

sensitivity/recall = true_positives / (true_positives + false_negatives)

precision = true_positives/(true_positives+false_positives)

Where am I going wrong, surely sklearn's classification problem can't be the problem, am I mis-reading something?

Edit: my function for calculating the precision and recall values given a confusion matrix from sklearn.metrics.confusion_matrix and a list of class numbers, for example for classes 1-3: [1, 2, 3] classes.

def calc_precision_recall(conf_matrix, class_labels):

    # for each class 
    for i in range(len(class_labels)):

        # calculate true positives
        true_positives =(conf_matrix[i, i])

        # false positives
        false_positives = (conf_matrix[i, :].sum() - true_positives)

        # false negatives
        false_negatives = 0
        for j in range(len(class_labels)):
            false_negatives += conf_matrix[j, i]
        false_negatives -= true_positives

        # and finally true negatives
        true_negatives = (conf_matrix.sum() - false_positives - false_negatives - true_positives)

        # print calculated values
        print(
            "Class label", class_labels[i],
            "T_positive", true_positives,
            "F_positive", false_positives,
            "T_negative", true_negatives,
            "F_negative", false_negatives,
            "\nSensitivity/recall", true_positives / (true_positives + false_negatives),
            "Specificity", true_negatives / (true_negatives + false_positives),
            "Precision", true_positives/(true_positives+false_positives), "\n"
        )

    return

Answer 1

Ok, where is your code? It's impossible to say for sure, when no one can see your code. I'll just take a stab here...maybe your data is imbalanced. Do you have more/less records in some feature columns? Resample arrays or sparse matrices in a consistent way.

This should run fine for you, right. Test it and see.

# Begin by importing all necessary libraries
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn import datasets


# import some data to play with
iris = datasets.load_iris()
X = iris.data[:, 0:3]  # we only take the first two features.
y = iris.target


# Now that we have the features and labels we want, we can split the data into training and testing sets using sklearn's handy feature train_test_split():

# Test size specifies how much of the data you want to set aside for the testing set. 
# Random_state parameter is just a random seed we can use.
# You can use it if you'd like to reproduce these specific results.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=27)

# You may want to print the results to be sure your data is being parsed as you expect:

print(X_train)  
print(y_train)

# Now we can instantiate the models. Let's try using two classifiers, a Support Vector Classifier and a K-Nearest Neighbors Classifier:

SVC_model = SVC()
# KNN model requires you to specify n_neighbors,
# the number of points the classifier will look at to determine what class a new point belongs to
KNN_model = KNeighborsClassifier(n_neighbors=5)

# Now let's fit the classifiers:

SVC_model.fit(X_train, y_train)
KNN_model.fit(X_train, y_train)

# The call has trained the model, so now we can predict and store the prediction in a variable:

SVC_prediction = SVC_model.predict(X_test)
KNN_prediction = KNN_model.predict(X_test)

#We should now evaluate how the classifier performed. There are multiple methods of evaluating a classifier's performance, and you can read more about there different methods below.

#In Scikit-Learn you just pass in the predictions against the ground truth labels which were stored in your test labels:

# Accuracy score is the simplest way to evaluate
print(accuracy_score(SVC_prediction, y_test))
print(accuracy_score(KNN_prediction, y_test))



# But Confusion Matrix and Classification Report give more details about performance
print(confusion_matrix(SVC_prediction, y_test))
print(classification_report(KNN_prediction, y_test))

Result:

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         7
           1       0.91      0.91      0.91        11
           2       0.92      0.92      0.92        12

    accuracy                           0.93        30
   macro avg       0.94      0.94      0.94        30
weighted avg       0.93      0.93      0.93        30

See the resources below.

https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets

https://scikit-learn.org/stable/modules/generated/sklearn.utils.resample.html

Oh, and the X an y variables both have 150 records.

X.shape
y.shape

Result:

X.shape
Out[107]: (150, 3)

y.shape
Out[108]: (150,)

Answer 2

I compared my returns per command and those made by hand and they both agree. I imagine you are wrongly considering the values (or some of the values) of TP, FN, FP, TN. It may help to look at a graph:

(Image taken from the internet: https://www.stardat.net/post/confusion-matrix )