How to improve perfomance on my cnn trained model

Question

my questions can sound really abstract but I couldn't find the solution to my problem in other discussions.

I have created my own dataset of pictures (128*128) to have a cnn model that recognizes speed limit sign. So I took around 2000 pictures approximately separated into 8 categories. So I have around 200 pictures or more for each sign.

I have tested different models simple ones, hard ones, did data augmentation to avoid overfitting but the result is always the same, I have "pretty bad" results or "noisy" results as you can see on the graph.

graph accuracy model 1

graph loss model 1

result model 1

graph accuracy model 2

graph loss model 2

result model 2

Here as you can see, I tested two different models with different parameters (different sizes of pictures, activations = elu, and kernel sizes different 8 and 16 to start).

So I'm more or less convinced of my work because when it comes from the real-time recognitions with my camera, it doesn't work well or the predictions are not always right (despite the 0.9 accuracies). Maybe it comes from my dataset that is not correct? My model? I'm asking a bit of light on this hard subject:).

I'll let you my model and an example of my dataset for the sign 20km/h.

example dataset 20km/h

This model code corresponds to the second graphs and result:

no_Of_Filters = 8
size_of_Filter = (3, 3)  # THIS IS THE KERNEL THAT MOVE AROUND THE IMAGE TO GET THE FEATURES.
# THIS WOULD REMOVE 2 PIXELS FROM EACH BORDER WHEN USING 100 100 IMAGE
size_of_pool = (2, 2)  # SCALE DOWN ALL FEATURE MAP TO GENERALIZE MORE, TO REDUCE OVERFITTING
no_Of_Nodes = 100  # NO. OF NODES IN HIDDEN LAYERS
model = Sequential()
# ADDING MORE CONVOLUTION LAYERS = LESS FEATURES BUT CAN CAUSE ACCURACY TO INCREASE
model.add((layers.Conv2D(no_Of_Filters, size_of_Filter, input_shape=(imageDimensions[0], imageDimensions[1], 1),
                         activation='elu')))

model.add((layers.Conv2D(no_Of_Filters * 2, size_of_Filter, activation='elu')))
model.add(layers.MaxPooling2D(pool_size=size_of_pool))

model.add((layers.Conv2D(no_Of_Filters * 4, size_of_Filter, activation='elu')))

model.add((layers.Conv2D(no_Of_Filters * 8, size_of_Filter, activation='elu')))
model.add(layers.MaxPooling2D(pool_size=size_of_pool))

model.add(Flatten())
model.add(Dense(no_Of_Nodes, activation='relu'))
# model.add(Dropout(0.2))  # INPUTS NODES TO DROP WITH EACH UPDATE 1 ALL 0 NONE
model.add(Dense(noOfClasses, activation='softmax'))  # OUTPUT LAYER
# COMPILE MODEL
model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])

Answer 1

There are a number of layers to your question so let me try to go step by step.

Firstly, your graphs are not "pretty bad" but rather are pretty fair indicators that your model is training. This is because the general trend of accuracy is upwards and loss is downwards (you can ignore the noise). In general the graphs should not have an increasing gap between the training and validation metrics because that is an indicator of overfitting.

This brings me to the next part - how have you split your training/validation/testing data? Is it the usual 60-20-20 split? If yes, is the split stratified (same proportion of each class in each of the splits)? Your testing data needs to be completely unseen data.

Finally, for actual inference using your camera, is the background setting same as the training images (which seem to be in an office setting). Testing on images captured on roads might throw the model off due to the addition detail in the background - especially as your model is a very simple one. It seems your task is some level of digit classification - for this you should take a look at models which have performed well for datasets like MNIST and SVHN. For the first step I'd suggest adding more convolutional layers and seeing if there is an improvement in performance. Additionally, you can try transfer learning using some of the inbuilt keras models which are pretrained on imagenet and therefore provide a good base for the training.