为什么 Keras 中 fit_generator 的精度、evaluate_generator 的精度和自定义精度不同？

Question

I wrote the following code:

epochs = 4
batch_size = 5
numclasses = 7

train_datagen = ImageDataGenerator(
    rescale=1. / 280,
    rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
    zoom_range = 0.1, # Randomly zoom image 
    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    #shear_range=0.2,
    vertical_flip=False,
    horizontal_flip=True)

# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1. / 280)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical')

nb_train_samples = 7*132
nb_validation_samples = 7*28

lr = 1e-5
decay = 1e-7 #0.0
optimizer = RMSprop(lr=lr, decay=decay)
# model is resnet
model.load_weights('modelweights.h5')
model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

history = model.fit_generator(
    train_generator,
    steps_per_epoch = len(train_generator),
    epochs=epochs,
    workers=1,
    use_multiprocessing=False,
    validation_data=validation_generator,
    validation_steps=len(validation_generator))

If I try to calculate the accuracy with evaluate_generator or with custom code, I get different accuracy from the one I got with fit_generator. More specifically:

y_true = validation_generator.classes
y_pred_test = model.predict_generator(validation_generator,verbose=1,steps=len(validation_generator))
predicted_class_indices=np.argmax(y_pred_test,axis=1)
custom_accuracy = accuracy_score(y_true,predicted_class_indices)

or

model.evaluate_generator(validation_generator,steps = len(validation_generator),verbose=1)

Accuracy variables are different values. Why is this happening?

Answer 1

generator.classes does not give you the labels in the same order than generator predictions, so any metric computed on this will not be correct.

The value computed by evaluate_generator is correct.

The only way to make a proper evaluation using a generator or Sequence is to iterate over all batches in the generator, computing the metric batch by batch (each batch has inputs and labels), and then aggregating the results.