Machine Learning model performs worse on test data than validation data

Question

I am quite new to machine learning.

To start things, I wanted to train a model to classify pictures of cats and dogs.

The problem I have is that when I train my model, it gives me a (approximately) 80-85% accuracy on the training data and the validation data. The loss is quite low with about 0.4 - 0.5 on both the validation data and the training data. Because those numbers are quite similar, I suspect that I don't have a problem with overfitting, right?

But when I test my model with pictures from the dataset (which it has not seen before), the accuracy turns out to be around 70%-73%. So it's significantly lower. I was not able to find any information on why this is the case. And, as I said, I suspect that overfitting is not the problem but since I'm a beginner, I'm not quite sure.

My model looks like this (I use tensorflow in python):

model = Sequential([
    Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(224,224,3), kernel_initializer="he_uniform"),
    MaxPool2D(pool_size=(2, 2)),
    Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding = 'same', kernel_initializer="he_uniform"),
    MaxPool2D(pool_size=(2, 2)),
    Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding = 'same', kernel_initializer="he_uniform", kernel_regularizer=l2(.001)),
    MaxPool2D(pool_size=(2, 2)),
    Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding = 'same', kernel_initializer="he_uniform", kernel_regularizer=l2(.001)),
    MaxPool2D(pool_size=(2, 2)),
    Flatten(),
    Dense(units = 128, activation='relu'),
    Dropout(.5),
    Dense(units=2, activation='softmax')
])

Trainable params: 3,471,810
Non-trainable params: 0

Optimizer, loss:

model.compile(optimizer=Adam(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])

This is the dataset, which I use: https://www.kaggle.com/tongpython/cat-and-dog I use 3000 images for training (1500 of dogs and 1500 of cats), 1000 for validation and 1000 for testing. There are no duplicates (so no images in the validation set, which are also in the training set and so on).

I preprocess the images like that (and I also use data augmentation):

train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input, rescale=1/255, horizontal_flip=True, vertical_flip=True, width_shift_range=.2, height_shift_range=.2) \
    .flow_from_directory(directory=training_path, target_size=(224,224), classes=['cat', 'dog'], batch_size=64)

valid_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input, rescale=1/255, horizontal_flip=True, vertical_flip=True, width_shift_range=.2, height_shift_range=.2) \
    .flow_from_directory(directory=validation_path, target_size=(224,224), classes=['cat', 'dog'], batch_size=64)

test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input).flow_from_directory(directory=test_path, target_size=(224,224), classes=['cat', 'dog'], batch_size=64, shuffle=False)

EDIT: Solved the problem. My mistake was that I did not preprocess the train, validation and test data in the exact same way because I misunderstood one parameter. Thank you to all, who helped me.

Answer 1

I believe your problem results from the fact that for the validation data and the training data you have

train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input, rescale=1/255, .....

The vgg16.preprocess_input function rescales the pixel value between +1 and -1 so there is no need to include rescale=1/255. In your test generator you do not rescale the pixel values. So remove the rescale=1/255 in the train and validation generators