Keras F1 score metrics for training the model

Question

I am new to keras and I want to train the model with F1-score as my metrics.

I came across two things, one is that I can add callbacks and other is using the in built metrics function Here , it says that the metrics function will not be used for training the model. So, does that mean I can anything in metrics argument while compiling the model? Specfically,

model.compile(optimizer='rmsprop',
    loss='binary_crossentropy',
    metrics=['accuracy'])

In the above case even though accuracy is passed as metrics, it will not be used for training the model.

Second thing is to use callbacks as defined here,

import numpy as np
from keras.callbacks import Callback
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
class Metrics(Callback):
def on_train_begin(self, logs={}):
 self.val_f1s = []
 self.val_recalls = []
 self.val_precisions = []

def on_epoch_end(self, epoch, logs={}):
 val_predict = (np.asarray(self.model.predict(self.model.validation_data[0]))).round()
 val_targ = self.model.validation_data[1]
 _val_f1 = f1_score(val_targ, val_predict)
 _val_recall = recall_score(val_targ, val_predict)
 _val_precision = precision_score(val_targ, val_predict)
 self.val_f1s.append(_val_f1)
 self.val_recalls.append(_val_recall)
 self.val_precisions.append(_val_precision)
 print “ — val_f1: %f — val_precision: %f — val_recall %f” %(_val_f1, _val_precision, _val_recall)
 return

metrics = Metrics()

Then fit the model,

model.fit(training_data, training_target, 
 validation_data=(validation_data, validation_target),
 nb_epoch=10,
 batch_size=64,
 callbacks=[metrics])

I am not sure if this will train the model on f1 score.

Answer 1

You can't train a neural network with f1-scores. For back propagating the error during training you need some sort of function which tells you, how far away your prediction is from the expected value. Such a function is as example the MSE loss.

F1 score on the other hand is just the harmonic mean between precision and recall from your samples. It does not tell you, in which direction you have to update the weights in order to get a better model. It also does not tell you, how far away you prediction is from the expected value.

What you could do is to print the F1 score after every epoch. An example on how to do this can be found in this blogpost