My CNN model accuracy is not improving with ResNet50

Question

I'm working on a machine learning project with TensorFlow for sign language recognition. When I first tried to build my CNN model it has a validation accuracy of 94% :

def create_model():  
 model = Sequential(name='SIGNS_LANGUAGE')
 model.add(layers.Conv2D(32,(3,3),activation='relu',input_shape=(IMG_SIZE, IMG_SIZE , 3) ) )
 model.add(layers.MaxPooling2D(pool_size = (2, 2)))
 model.add(layers.Conv2D(64, (3, 3), activation='relu' ,input_shape=(IMG_SIZE, IMG_SIZE , 3) ) )
 model.add(layers.MaxPooling2D(pool_size = (2, 2)))
 #classification layers 
 model.add(layers.Flatten())
 model.add(layers.Dense(512, activation='relu'))
 model.add(layers.Dropout(0.2))
 model.add(layers.Dense(512, activation='relu'))
 model.add(layers.Dropout(0.2))
 model.add(layers.Dense(512, activation='relu'))
 model.add(layers.Dropout(0.2))
 model.add(layers.Dense(targetCount, activation='softmax'))
 print(model.summary())
 return model`

BUT IT HASN'T BEEN GOOD AT generalizing with a different data.

So I tried to work with a pre-trained model like ResNet50 but I couldn't figure out why the accuracy is not improving with time, it hit 40% and stopped:

def create_model():
  model = Sequential(name='SIGNS')
  input_shape = (IMG_SIZE , IMG_SIZE , 3)
  res_layer = tf.keras.applications.ResNet50(input_shape=input_shape, include_top=False , weights='imagenet' )
  res_layer.trainable = False
  model.add( res_layer )
  model.add(layers.GlobalAveragePooling2D())
  model.add(layers.Dropout(0.2))
  model.add(layers.Dense(1024, activation='relu'))
  model.add(layers.Dropout(0.2))
  model.add(layers.Dense(512, activation='relu'))
  model.add(layers.Dropout(0.2))
  model.add(layers.Dense(targetCount, activation='softmax'))
  print(model.summary())
  return model

Any suggestions on how to improve the model??

NB: I input 54049 grayscale images in the 3 channels and I use Adam optimizer.

Answer 1

If you are using a pre-trained architecture, you have to normalize your data according to the dataset on which the pre-trained architecture was originally trained on(in this case, Imagenet).

Try incorporating this code after your first one -

preprocessing_layer = tf.keras.applications.resnet50.preprocess_input(x, data_format=None)
inputs = keras.Input(shape=input_shape)
x = preprocessing_layer(inputs)
outputs = model(x)
inference_model = keras.Model(inputs, outputs)