Error in layers while loading weights from a pretrained model in keras for finetuning

Question

I am trying to follow the tutorial Fine-tuning the top layers of a pre-trained network .

For this purpose I want to use the pretrained keras-facenet and add my classifier on top. I am using vggface as the base model. Facenet is based on VGGFace. So here is the error which I get after running my code:

ValueError                                Traceback (most recent call last)
<ipython-input-9-261fed5d7ddc> in <module>()
     20 model.add(layers.Dense(12, activation='sigmoid'))
     21 
---> 22 model.load_weights(top_model_weights_path)
     23 
     24 

/usr/local/lib/python3.6/dist-packages/keras/models.py in load_weights(self, filepath, by_name, skip_mismatch, reshape)
    766                                                               reshape=reshape)
    767             else:
--> 768                 topology.load_weights_from_hdf5_group(f, layers, reshape=reshape)
    769 
    770     def save_weights(self, filepath, overwrite=True):

/usr/local/lib/python3.6/dist-packages/keras/engine/topology.py in load_weights_from_hdf5_group(f, layers, reshape)
   3363                          'containing ' + str(len(layer_names)) +
   3364                          ' layers into a model with ' +
-> 3365                          str(len(filtered_layers)) + ' layers.')
   3366 
   3367     # We batch weight value assignments in a single backend call

ValueError: You are trying to load a weight file containing 245 layers into a model with 2 layers.

Here is the code:

# path to the model weights files.
weights_path = 'keras-facenet/weights/facenet_keras_weights.h5'
top_model_weights_path = 'keras-facenet/model/facenet_keras.h5'
# dimensions of our images.
img_width, img_height = 224, 224

train_data_dir = 'dataset_cfps/train'
validation_data_dir = 'dataset_cfps/validation'
nb_train_samples = 1774
nb_validation_samples = 313
epochs = 50
batch_size = 16

vggface = VGGFace(model='resnet50', include_top=False, input_shape=(img_width, img_height, 3))

# Create the model
model = models.Sequential()
model.add(layers.Flatten( input_shape=vggface.output_shape[1:]))
model.add(layers.Dense(256, activation='relu'))
model.add(layers.Dropout(0.5))
model.add(layers.Dense(12, activation='sigmoid'))

model.load_weights(top_model_weights_path)

custom_vgg_model = Model(vggface.input, model(vggface.output))

for layer in custom_vgg_model.layers[:-3]:
    layer.trainable = False

custom_vgg_model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

# prepare data augmentation configuration
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='categorical')

custom_vgg_model.summary()

# fine-tune the model
custom_vgg_model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    verbose=2)

# Save the model
custom_vgg_model.save('facenet_latest_lr4.h5')

What can be the problem for the error? Is it difference in the dimension of the layers between the pretrained and the classification model?

Answer 1

I had this error before, albeit on another data set and architecture. The problem is that the topology is different. You can try the following code:

model.load_weights('filename.h5' , by_name = True, skip_mismatch = True) 

It will only load layers that correspond in the number of weights.

Documentation is in topology.py

Answer 2

First of all, you try to load the weights into your classifier model, not the VGGFace model. In your code, you would have to write vggface.load_weights(top_model_weights_path) instead of model.load_weights(top_model_weights_path) . Assuming both models are equal in architecture, this should work. But why do you even do this in such a complicated way? VGGFace seems to provide a simple and convinient way to (down)load pre-trained weights by passing weights='vggface' when creating VGGFace . Except you really want to use the weights from keras-facenet, you should go with the described way.

Furthermore it doesn't look like that the VGGFace model is even connected with your custom classifier. I'm not sure if this can be achieve using a Sequential model. Probably you would need to use the functional API and call the first layers of your classifier with vggface .

Also note that with model(vggface.output) your calling the VGGFace model with your custom model as input and not the opposite like it was probably intended.