Deep learning CNN model not learning

Question

I am working on a image classification project and my model doesn't seem to train properly.

My dataset is made of 4000 images each with a shape of (120,120,3). Test set represents 20% of the total dataset. All images have been correctly labeled. The images are normalized and one-hot encoded. For now I use only two targets, but I will add one more one I start getting decent results. I use a batch size of 16 I want to use a CNN model.

My current model:

model = keras.models.Sequential()
model.add(Conv2D(filters=16, kernel_size=(6,6), input_shape=(IMG_SIZE,IMG_SIZE,3), activation='relu',))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.2))

model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu',))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(filters=64, kernel_size=(4,4), activation='relu',))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(filters=128, kernel_size=(3,3), activation='relu',))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.2))
model.add(Conv2D(filters=256, kernel_size=(2,2), activation='relu',))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.2))


model.add(Flatten())


model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))

model.compile(optimizer='nadam',
             loss='categorical_crossentropy',
            metrics=['accuracy'])

model.summary()

model summary gives: Total params: 273,330 Trainable params: 273,330 Non-trainable params: 0

from tensorflow.keras.callbacks import EarlyStopping
early_stop = EarlyStopping(monitor='val_loss',patience=10)



history = model.fit(x_train_sample, y_train_sample,
                   batch_size = BATCH_SIZE,
                   epochs = EPOCHS,
                   verbose = 1,
                   validation_data = (x_test, y_test)
                   ,callbacks=[early_stop,PlotLossesKeras()])

When I run my model for 30 epochs, earlystopping triggers.

Epoch 1/30
43/43 [==============================] - 9s 205ms/step - loss: 0.1109 - accuracy: 0.9531 - val_loss: 0.5259 - val_accuracy: 0.8397
Epoch 2/30
43/43 [==============================] - 10s 231ms/step - loss: 0.0812 - accuracy: 0.9692 - val_loss: 0.5793 - val_accuracy: 0.8355
Epoch 3/30
43/43 [==============================] - 9s 219ms/step - loss: 0.1000 - accuracy: 0.9721 - val_loss: 0.5367 - val_accuracy: 0.8547
Epoch 4/30
43/43 [==============================] - 9s 209ms/step - loss: 0.0694 - accuracy: 0.9707 - val_loss: 0.6101 - val_accuracy: 0.8269
Epoch 5/30
43/43 [==============================] - 9s 203ms/step - loss: 0.0891 - accuracy: 0.9633 - val_loss: 0.6116 - val_accuracy: 0.8419
Epoch 6/30
43/43 [==============================] - 9s 210ms/step - loss: 0.0567 - accuracy: 0.9765 - val_loss: 0.4833 - val_accuracy: 0.8419
Epoch 7/30
43/43 [==============================] - 9s 218ms/step - loss: 0.0312 - accuracy: 0.9897 - val_loss: 1.4513 - val_accuracy: 0.8034
Epoch 8/30
43/43 [==============================] - 9s 213ms/step - loss: 0.0820 - accuracy: 0.9707 - val_loss: 0.5821 - val_accuracy: 0.8248
Epoch 9/30
43/43 [==============================] - 9s 222ms/step - loss: 0.0513 - accuracy: 0.9897 - val_loss: 0.8516 - val_accuracy: 0.8462
Epoch 10/30
43/43 [==============================] - 11s 246ms/step - loss: 0.0442 - accuracy: 0.9853 - val_loss: 0.7927 - val_accuracy: 0.8397
Epoch 11/30
43/43 [==============================] - 10s 222ms/step - loss: 0.0356 - accuracy: 0.9897 - val_loss: 0.7730 - val_accuracy: 0.8141
Epoch 12/30
43/43 [==============================] - 10s 232ms/step - loss: 0.0309 - accuracy: 0.9824 - val_loss: 0.9528 - val_accuracy: 0.8226
Epoch 13/30
43/43 [==============================] - 9s 220ms/step - loss: 0.0424 - accuracy: 0.9839 - val_loss: 1.2109 - val_accuracy: 0.8013
Epoch 14/30
43/43 [==============================] - 10s 228ms/step - loss: 0.0645 - accuracy: 0.9824 - val_loss: 0.5308 - val_accuracy: 0.8547
Epoch 15/30
43/43 [==============================] - 11s 259ms/step - loss: 0.0293 - accuracy: 0.9927 - val_loss: 0.9271 - val_accuracy: 0.8333
Epoch 16/30
43/43 [==============================] - 9s 217ms/step - loss: 0.0430 - accuracy: 0.9795 - val_loss: 0.6687 - val_accuracy: 0.8483

I have tried many different model architectures, changing number of layers, kernel size etc... I can't seem to figure out what is going wrong.

Answer 1

There are many possible reasons.

For starters, depending on your categories, you might want to consider using transfer learning to speed up your training process.

Your architecture looks reasonable and the training and validation loss seems right as well (overfitting is occurring).

Given that you've stated that you could have 3 categories and am currently only using 2, might there be a different distribution between your training set and your test set? That might be causing the model to be unable to generalise well.

For instance, your dataset contains of evenly distributed number of images of Cats, Dogs and Humans. You set 2 categories to train on and thus your model attempts to segment between humans and animals when it tries to validate, there is an uneven distribution in the training data causing the model to see insufficient training size of humans (33%)?