How to fo transfer learning of a resnet50 model with with own dataset?

Question

I am trying to build a face verification system using keras and resnet50 model with vggface weights. The way i am trying to achieve this is by the following steps:

• given two image i first find out the face using mtcnn as embeddings
• then i calculate the cosine distance between two vector embeddings. the distance starts from 0 to 1..... (Here to be noted that the lower the distance the same two faces is)

Using the pre-trained model of resnet50 i get fairly good result. But since the model was trained mostly on european data and i want face verification on indian sub-contient i cannot rely on that. I want to train them on my own dataset. I have 10000 classes with each class containing 2 image. With image augmentation i can create 10-15 image per class from those two image.

here is the sample code i am using for training

base_model = VGGFace(model='resnet50',include_top=False,input_shape=(224, 224, 3))
base_model.layers.pop()
base_model.summary()
for layer in base_model.layers:
    layer.trainable = False


y=base_model.input
x=base_model.output
x=GlobalAveragePooling2D()(x)
x=Dense(1024,activation='relu')(x) #we add dense layers so that the model can learn more complex functions and classify for better results.
x=Dense(1024,activation='relu')(x) #dense layer 2
x=Dense(512,activation='relu')(x) #dense layer 3
preds=Dense(8322,activation='softmax')(x) #final layer with softmax activation

model=Model(inputs=base_model.input,outputs=preds)


model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])

model.summary()
train_datagen=ImageDataGenerator(preprocessing_function=preprocess_input) #included in our dependencies

train_generator=train_datagen.flow_from_directory('/Users/imac/Desktop/Fayed/Facematching/testenv/facenet/Dataset/train', # this is where you specify the path to the main data folder
                                                 target_size=(224,224),
                                                 color_mode='rgb',
                                                 batch_size=32,
                                                 class_mode='categorical',
                                                 shuffle=True)
step_size_train=train_generator.n/train_generator.batch_size


model.fit_generator(generator=train_generator,
                   steps_per_epoch=step_size_train,
                   epochs=10)
model.save('directory')

As far as the code code is concern what i understand is that i disable the last layer then add 4 layer train them and store them in a diectory.

i then load the model using

model=load_model('directory of my saved model')
model.summary()
yhat = model.predict(samples)

i predict the embedding of two image and then calculate cosine distance. But the problem is that the prediction gets worsen with my trained model. For two image of same person the pre-trained model gives distance of 0.3 whereas my trained model show distance of 1.0. Although during training loss function is decreasing with each epoch and accuracy is increasing but that doesn't reflect on my prediction output. I want to increase the prediction result of pre-trained model.

How can i achieve that with my own data?

NB: I am relatively new in machine learning and don't know a lot about model layers

Answer 1

What I would suggest is to go with triplet or siamese with these many number of classes. Use MTCNN to extract faces and then use facenet architecture to generate 512 dimensions embedding vectors, then visualize it using TSNE plot. Every face will be assigned a small embedding cluster. Go through this link for Keras to generate face embeddings: Link .

Then, try Triplets semi-hard and hard loss on your dataset to cluster them into 10000 classes. It might help. Go through this detailed blog on triplet loss: Triplets . Codes to go through some of the repositries: Code .