Keras model.evaluate_generator not printing expected loss (and accuracy?)

Question

Problem

I want to increase the image size from 64 to 128 to see how that would affect my model. However, this makes the dataset too large to fit into memory. Therefore, I tried to switch to using Keras generators to fit my model while using image size 64 to make sure my implementation is correct. model.fit_generator seems to be working as expected when comparing the train and validation logs to model.fit 's logs, but model.evaluate_generator is not showing expected results when compared to the model.evaluate output.

The loaded model was trained using model.fit . I don't know if this affects anything. There are 5000 validation imgs for validation_generator (chosen by ImageDataGenerator) and 5000 randomly selected validation imgs in X_val and Y_val

Using model.evaluate_generator

IMG_SIZE = 64
BATCH_SIZE = 32
EPOCHS = 30
datagen = ImageDataGenerator(validation_split=.2)
train_generator = datagen.flow_from_directory(directory=TRAIN_DIR, target_size=(IMG_SIZE, IMG_SIZE), class_mode='categorical', batch_size=BATCH_SIZE, shuffle=True, subset='training')
validation_generator = datagen.flow_from_directory(directory=TRAIN_DIR, target_size=(IMG_SIZE, IMG_SIZE), class_mode='categorical', batch_size=BATCH_SIZE, shuffle=False, subset='validation')
predict_generator = datagen.flow_from_directory(directory=TEST_DIR, target_size=(IMG_SIZE, IMG_SIZE), class_mode=None, batch_size=1, shuffle=False)

Evaluating:

cat_dog_model = keras.models.load_model('model/'+MODEL_NAME)
results = cat_dog_model.evaluate_generator(generator=validation_generator, steps=validation_generator.samples//BATCH_SIZE, verbose=1)

evaluate_generator outputs this:

# with validation_generator batch_size=32
156/156 [==============================] - 18s 118ms/step - loss: 10.3093 - accuracy: 0.7957
# with validation_generator batch_size=1
5000/5000 [==============================] - 36s 7ms/step - loss: 10.3888 - accuracy: 0.7952

Using model.evaluate

Evaluating:

results = cat_dog_model.evaluate(X_val, Y_val, batch_size=32, verbose=1)

evaluate output:

157/157 [==============================] - 15s 96ms/step - loss: 0.2297 - accuracy: 0.9066

Answer 1

It is a bit hard to tell without the complete code and data. However, it seems like the input is indeed different. You say

There are 5000 validation imgs for validation_generator (chosen by ImageDataGenerator) and 5000 randomly selected validation imgs in X_val and Y_val

And I can see that the input variables are indeed different, X_val, Y_val vs validation_generator.

However, please note that evaluate (and same for fit) is able to handle generators. There is no need for fit on generator and evaluate generator. Actually you should have gotten a warning that these functions are deprecated. You can use:

results = cat_dog_model.evaluate(validation_generator)

or

results = cat_dog_model.evaluate(validation_generator, steps=validation_generator.samples//BATCH_SIZE, verbose=1)