generating confusion matrix in keras for multiclass classification

Question

Getting accuracy up to 98% by training model but confusion matrix shows very high miss-classification.

I am working on multiclass classification using keras with transfer learning approach on pre-trained VGG16 model.

The problem is to classify the images into 5 types of tomato diseases using CNN.

There are 5 disease classes with 6970 training images and 70 testing images.

Training model shows 98.65% accuracy while testing shows 94% accuracy.

But the problem is when I am generating confusion matrix it shows very high miss-classification.

someone please help me, whether my code is wrong or the model is wrong? I am confused whether my model is giving me correct results or not.

And if someone can explain me how keras actually calculate the accuracy using model.fit_generator Function because applying the general formula of accuracy on confusion matrix is not giving me same results as keras have calculated.

For testing the dataset code is:

test_generator = test_datagen.flow_from_directory(
test_dir,
target_size=(150, 150),
batch_size=20,
class_mode='categorical')
test_loss, test_acc = model.evaluate_generator(test_generator, steps=50)
print('test acc:', test_acc)

I found the code to generate confusion matrix from one of the forum;

code is:

import numpy as np
from sklearn.metrics import confusion_matrix,classification_report
batch_size = 20
num_of_test_samples = 70
predictions = model.predict_generator(test_generator,  num_of_test_samples // batch_size+1)

y_pred = np.argmax(predictions, axis=1)

true_classes = test_generator.classes

class_labels = list(test_generator.class_indices.keys())   

print(class_labels)

print(confusion_matrix(test_generator.classes, y_pred))

report = classification_report(true_classes, y_pred, target_names=class_labels)
print(report)

Following are the results I get:

Testing accuracy:

Found 70 images belonging to 5 classes.
test acc: 0.9420454461466182

Results of Confusion matrix:

['TEB', 'TH', 'TLB', 'TLM', 'TSL']
[[2 3 2 4 3]
 [4 2 3 0 5]
 [3 3 3 2 3]
 [3 3 2 4 2]
 [2 2 4 4 2]]]
              precision    recall  f1-score   support

         TEB       0.14      0.14      0.14        14
          TH       0.15      0.14      0.15        14
         TLB       0.21      0.21      0.21        14
         TLM       0.29      0.29      0.29        14
         TSL       0.13      0.14      0.14        14

   micro avg       0.19      0.19      0.19        70
   macro avg       0.19      0.19      0.19        70
weighted avg       0.19      0.19      0.19        70

Answer 1

While creating the test data generator, the flow_from_directory method takes shuffle=True parameter by default. Therefore, when you predict by plugging in the generator instance, the predictions are not shown in the same order as the true classes are. This is the reason you are getting the right predictions, but in a different order. So, the confusion matrix is showing bad performance.

Just set shuffle to False in the test data generator and the predictions will come in the right order. As the purpose of validation/test data is to evaluate the model, you can almost always set shuffle to False.

Answer 2

The test labels should be class_indices rather than classes

true_classes = test_generator.class_indices

Answer 3

Dear always do the following for any classification performance parameters:

1. first reset the generator which you are using in prediction
2. put shuffle equal to false in flow_from_directory()

Answer 4

I may be late to the party but maybe you aren't preprocessing test data the same way as train ones. Try to import preprocessing function from VGG16 and add it to generator as parameter (preprocessing_function).