50% validation accuracy in binary classification with any CNN model (including VGG16)

Question

I'm very new to CNNs and I'm having an issue of my binary classifcation generating a validation accuracy of 50%.

For the record, I'm attempting to catgegorise MRIs of brain scans into Alzheimer's Disease, and healthy control.
The images by default are 250x250px in black and whiite.png format, and I have roughly 1,000 images total.
I have tried creating my own models, and implementing VGG16, all of which produde a validation accuracy of around 50%.

I'm starting to think it might be an issue of the data being processed incorrectly but as I'm new to this I'm unsure. Maybe something to do with the images being interpreted as RGB when they're black and white? But I'm honestly not sure.

Would appreciate anyone taking a look, thank you!

import tensorflow as tf
from keras.layers.convolutional import Convolution2D
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense, Flatten, BatchNormalization, Conv2D, MaxPool2D, Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import os
import warnings
import matplotlib.pyplot as plt
from tensorflow.python.keras.applications.vgg16 import VGG16
from tensorflow.python.keras.layers import ZeroPadding2D, MaxPooling2D

warnings.simplefilter(action='ignore', category=FutureWarning)

os.chdir('C:/Users/dancu/PycharmProjects/firstCNN/data/ad-vs-cn')

physical_devices = tf.config.experimental.list_physical_devices('GPU')
print("Num GPUs Available: ", len(physical_devices))
tf.config.experimental.set_memory_growth(physical_devices[0], True)

# Define paths for image data
train_path = "C:/Users/dancu/PycharmProjects/firstCNN\data/ad-vs-cn/train"
test_path = "C:/Users/dancu/PycharmProjects/firstCNN\data/ad-vs-cn/test"
valid_path = "C:/Users/dancu/PycharmProjects/firstCNN\data/ad-vs-cn/valid"

# Use ImageDataGenerator to create 3 lots of batches
train_batches = ImageDataGenerator(
    rescale=1/255).flow_from_directory(directory=train_path,
        target_size=(64,64), classes=['cn', 'ad'], batch_size=20,
            color_mode="rgb")
valid_batches = ImageDataGenerator(
    rescale=1/255).flow_from_directory(directory=valid_path,
        target_size=(64,64), classes=['cn', 'ad'], batch_size=20,
            color_mode="rgb")
test_batches = ImageDataGenerator(
    rescale=1/255).flow_from_directory(directory=test_path,
        target_size=(256,240), classes=['cn', 'ad'], batch_size=10,
            color_mode="rgb")

imgs, labels = next(train_batches)

# Test to see normalisation has occurred properly
print(imgs[1][16])
print(labels)

# Define method to plot MRIs
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 10, figsize=(20,20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()

# Plot a sample of MRIs
plotImages(imgs)

# Define the model
# VGG16
model = Sequential()
model.add(Conv2D(input_shape=(64,64,3),filters=64,kernel_size=(3,3),padding="same", activation="relu"))
model.add(Conv2D(filters=64,kernel_size=(3,3),padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))
model.add(Flatten())
model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=2, activation="softmax"))

# This model hits around 70% train acc, 50% val acc
# model = Sequential([
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(64,64,3)),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.2),
#   #  BatchNormalization(),
#     Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.3),
#    # BatchNormalization(),
#     Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.4),
#   #  BatchNormalization(),
#     Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.4),
#     Flatten(),
#     Dense(units=2, activation='softmax')
# ])

## This model hits around 68% training accuracy at it's peak
# base_model = Sequential([
#     Conv2D(filters=32, kernel_size=(3, 3), activation='relu', padding = 'same', input_shape=(256,256,3)),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.1),
#     Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.2),
#     Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same'),
#     MaxPool2D(pool_size=(2, 2), strides=2),
#     Dropout(0.3),
#     Flatten(),
#     Dense(units=2, activation='softmax')
# ])

# Summarise each layer of the model
print(model.summary())

# Compile and train the model
model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x=train_batches,
    steps_per_epoch=len(train_batches),
    validation_data=valid_batches,
    validation_steps=len(valid_batches),
    epochs=35,
    verbose=1
)

EDIT: Thank you all for your replies so far, they've all been really insightful. I think we're reaching the conclusion that it's a case of small sample size and the fact that these images are hard to classify using 2D CNNs. Tomorrow I'm going to try and throw together a basic 3D CNN using the original.nii files and see if this increases the accuracy.

Answer 1

Repeat all images 3 times to conform to RGB 3 channels yourself. In loss, use binary_crossentropy . In the test image generator, you have (256, 240), make it the same as your training size.

Also, try out the original size for the network (224x224).

Answer 2

You have only 2 classes: Yes and No. Therefore, I would recommend to generate the output of one channel only with "sigmoid" activation, ie the last layer of your network should be:

Dense(units=1, activation='sigmoid')

and than train your network with "binary_crossentropy". Use the small simple CNN for the beginning.

Answer 3

In the training and validation generators you have made the image size only 64 X 64 versus the original size of 250 X 250. Reducing the image losses detail that the network can learn on. I recommend you set these values to 224 X 224. You have 1000 images which may not be enough to get a high level of accuracy. You may want to augment the data by using the augmentations available in the image data generator. Documentation is here. In your test generator you should make the image size the same size as was used for training. When you use VGG16 be sure to set the weights=imagenet and the image size to 224 X 224. You are doing a binary classification so I recommend top layer be Dense(units=1, activation='sigmoid'). Then you can just test if the ouput is above or below.5. When you compile the model use loss="binary_crossentropy". Also set top=False and pooling=max, Then add the dense layer. I don't use Vgg16 because it has over 40 million trainable parameters. I you train the full model the training time can be very long. I prefer to use Mobilenet which has only 4 million parameters and is essentially just as accurate.