为图像分类模型绘制混淆矩阵

Question

I built an image classification CNN with keras. While the model itself works fine (it is predicting properly on new data), I am having problems plotting the confusion matrix and classification report for the model.

I trained the model using ImageDataGenerator

train_path = '../DATASET/TRAIN'
test_path = '../DATASET/TEST'
IMG_BREDTH = 30
IMG_HEIGHT = 60
num_classes = 2

train_batch = ImageDataGenerator(featurewise_center=False,
                                 samplewise_center=False, 
                                 featurewise_std_normalization=False, 
                                 samplewise_std_normalization=False, 
                                 zca_whitening=False, 
                                 rotation_range=45, 
                                 width_shift_range=0.2, 
                                 height_shift_range=0.2, 
                                 horizontal_flip=True, 
                                 vertical_flip=False).flow_from_directory(train_path, 
                                                                          target_size=(IMG_HEIGHT, IMG_BREDTH), 
                                                                          classes=['O', 'R'], 
                                                                          batch_size=100)

test_batch = ImageDataGenerator().flow_from_directory(test_path, 
                                                      target_size=(IMG_HEIGHT, IMG_BREDTH), 
                                                      classes=['O', 'R'], 
                                                      batch_size=100)

This is the code for the confusion matrix and classification report

batch_size = 100
target_names = ['O', 'R']
Y_pred = model.predict_generator(test_batch, 2513 // batch_size+1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
cm = metrics.confusion_matrix(test_batch.classes, y_pred)
print(cm)
print('Classification Report')
print(metrics.classification_report(test_batch.classes, y_pred))

for the confusion matrix I get the rolling result (which seems to be wrong)

Confusion Matrix
[[1401    0]
 [1112    0]]

The False positives and true positives are 0. For the classification report I get this following output and warning

Classification Report
             precision    recall  f1-score   support

          0       0.56      1.00      0.72      1401
          1       0.00      0.00      0.00      1112

avg / total       0.31      0.56      0.40      2513

/Users/sashaanksekar/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)

I am trying to predict if an object is organic or recyclable. I have around 22000 train images and 2513 test images.

I am new to machine learning. what am I doing wrong?

Thanks in advance

Answer 1

To plot the confusion matrix do the following:

import matplotlib.pyplot as plt
import numpy as np

cm = metrics.confusion_matrix(test_batch.classes, y_pred)
# or
#cm = np.array([[1401,    0],[1112, 0]])

plt.imshow(cm, cmap=plt.cm.Blues)
plt.xlabel("Predicted labels")
plt.ylabel("True labels")
plt.xticks([], [])
plt.yticks([], [])
plt.title('Confusion matrix ')
plt.colorbar()
plt.show()

References:

https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

https://machinelearningmastery.com/confusion-matrix-machine-learning/

Answer 2

If someone got here like me because of similar issue there may be several things that could help:

1. Make sure you set shuffle = False in your test set generator;
2. It's better to set the batch_size to a divisor of your image count. If not - make sure the generator doesn't skip any images;
3. Try training without augmentation first;

4. There seems to be an issue where output of the predict_generator is not consistent, try setting workers = 0 if possible, like this:

   predictions = model.predict_generator(testGenerator, steps = np.ceil(testGenerator.samples / testGenerator.batch_size), verbose=1, workers=0)

In my case the predictions changed each time I called predict_generator if I didn't do it.

6. When you have only two classes you have to use:

   predictedClasses = np.where(predictions>0.5, 1, 0) instead of np.argmax(Y_pred, axis=1) since in this case np.argmax will always output 0.

   np.where(predictions>0.5, 1, 0) returns 1 if prediction > 0.5 else returns 0.

Answer 3

I use sklearn plot_confusion_matrix

To use it I made a hack so when the sklearn estimator makes prediction dont complaints because is a Keras model. So, if model is a trained keras model:

X,y = test_generator.next()
y = np.argmax(y, axis=1)

from sklearn.metrics import plot_confusion_matrix
class newmodel(MLPClassifier):
    def __init__(self, model):
        self.model = model
    def predict(self, X):
        y = self.model.predict(X)
        return np.argmax(y,axis=1)

model1 = newmodel(model)
plot_confusion_matrix(model1, X, y , normalize='true', xticks_rotation = 'vertical', display_labels = list(train_generator.class_indices.keys()))

It works for me.