High accuracy on both training and validation but very low on test set

Question

My CNN model has about 96~97% accuracy on both training and validation sets. But when submitting the test set it got only 24% accuracy. Here's my model:

def build_cnn_model():
    classifier = Sequential()
    classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 3), activation='relu'))
    classifier.add(MaxPooling2D())
    classifier.add(Convolution2D(32, (3, 3), activation='relu'))
    classifier.add(MaxPooling2D())
    classifier.add(Convolution2D(64, (3, 3), activation='relu'))
    classifier.add(MaxPooling2D())
    classifier.add(Flatten())
    classifier.add(Dense(64, activation='relu'))
    classifier.add(Dropout(0.5))
    classifier.add(Dense(4, activation='softmax'))

    classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

    return classifier

The training set has about 40k images, valid set has about 10k images and test set is made of 5.5k images. Here's my implementation

train_datagen = ImageDataGenerator(rescale=1./255)
valid_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory(
    'datasets/training_set',
    target_size=(64, 64),
    batch_size=32,
    seed=42,
    class_mode='categorical')

valid_set = valid_datagen.flow_from_directory(
    'datasets/valid_set/',
    target_size=(64, 64),
    batch_size=32,
    seed=42,
    class_mode='categorical')

test_set = test_datagen.flow_from_directory(
    'original_data/',
    classes=['test'],
    target_size=(64, 64),
    seed=42,
    class_mode=None,
    batch_size=1)

test_set.reset()

classifier = build_cnn_model()

classifier.fit_generator(
    training_set,
    epochs=10,
    steps_per_epoch=1222,
    validation_data=valid_set,
    validation_steps=305)

Here we can see the behavior of the model during training and I noticed that validation accuracy is always higher than training accuracy. So why this is happening? Why so low accuracy on test set and why validation accuracy is higher than training accuracy? What are the possible solutions?

Answer 1

You may be doing something funky with the labels on the test set that isn't happening to the train/validation.

Why are you passing classes=["test"] to the test datagen but not the others?

From documentation : "classes: Optional list of class subdirectories (eg ['dogs', 'cats']). Default: None. If not provided, the list of classes will be automatically inferred from the subdirectory names/structure under directory, where each subdirectory will be treated as a different class (and the order of the classes, which will map to the label indices, will be alphanumeric). The dictionary containing the mapping from class names to class indices can be obtained via the attribute class_indices."

Does the 'original_data/' have the same folder structure that 'datasets/training_set' and 'datasets/valid_set/' have?