CNN giving random answers while fully-connected neural network works fine

Question

I'm working on an AI that should analyze images and recognize 5 possible objects. I am using tf.keras in python, which worked fine on a previous project, but for this one it gives random results (20% accuracy with 5 possible outputs), no matter how long I train it.

The 5 possible objects are : - Four-legged animals - Humain figures - Airplanes - Trucks - Cars

I tried with a fully-connected neural network before, with the exact same data set, and it worked with a ~50% accuracy with 10 minutes of training. My goal is to reach 90% accuracy, that's why I'm trying to use a CNN instead. I also played a bit with the mnist dataset and made a cnn that recognize hand-written digits, using tf.keras. I tried to use the same keras model, but it failed. I tried different models, and all of them failed to give non-random predictions.

Here is my keras model :

import tensorflow as tf

layers = tf.keras.layers

model = tf.keras.models.Sequential([
    layers.Conv2D(16, (3, 3), input_shape=(80, 80, 1), activation='relu'),
    layers.MaxPooling2D(pool_size=(2, 2)),
    layers.Conv2D(32, (3, 3), activation='relu'),
    layers.MaxPooling2D(pool_size=(2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D(pool_size=(2, 2)),
    layers.Dropout(0.2),
    layers.Flatten(),
    layers.Dense(512, activation=tf.nn.relu),
    layers.Dense(5, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy']
)

And I'm training it with this piece of code :

x, y = self.load_data(input("File containg train files: "), input("File containg labels files: ")) # function that takes a .mat file and return an array of shape (29160, 1, 80, 80)

x = x.reshape(total_data_number, 80, 80, 1)
x = x.astype('float32')
x /= 255.0

epoch = 15

model.fit(x, y, batch_size=50, epochs=epoch, verbose=1)

The fully-connected model was this one :

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(80, 80, 1)),
tf.keras.layers.Dense(512, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(5, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

The accuracy should be way higher than 20%, and should improve when I train the model.

Answer 1

Some of your problem may come from the fact that you are using sparse_categorical_crossentropy whereas your output is a one_hot_encoded vector. So you should use categorical_crossentropy instead. (basic explanation here )

Answer 2

How many data do you have for each classes? Beside, your model seems too simple for an object classification task. You should try deeper models. Keras provides some pretrained models on imagenet . You may finetune those models on your own dataset. For example on Keras Application it provides a way to finetune a pretrianed InceptionV3 model:

from keras.applications.inception_v3 import InceptionV3
from keras.preprocessing import image
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)

# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# let's add a fully-connected layer
x = Dense(1024, activation='relu')(x)
# and a logistic layer -- 4 classes in your case
predictions = Dense(4, activation='softmax')(x)

# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)

# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
    layer.trainable = False

# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')

# train the model on the new data for a few epochs
model.fit_generator(...)

# at this point, the top layers are well trained and we can start fine-tuning
# convolutional layers from inception V3. We will freeze the bottom N layers
# and train the remaining top layers.

# let's visualize layer names and layer indices to see how many layers
# we should freeze:
for i, layer in enumerate(base_model.layers):
   print(i, layer.name)

# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:249]:
   layer.trainable = False
for layer in model.layers[249:]:
   layer.trainable = True

# we need to recompile the model for these modifications to take effect
# we use SGD with a low learning rate
from keras.optimizers import SGD
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy')

# we train our model again (this time fine-tuning the top 2 inception blocks
# alongside the top Dense layers
model.fit_generator(...)