How can I calculate precision, recall and F1-score in Neural Network models?

Question

I'm using Keras to predict if I'll get an output of 1 or 0. The data looks like this:

    funded_amnt  emp_length  avg_cur_bal  num_actv_rev_tl    loan_status
    10000       5.60088      19266                 2                  1
    13750       5.60088      2802                  6                  0
    26100       10.0000      19241                17                  1

The target is loan_status and the features are the remaining. I've normalized the data before starting to build a Neural Network model.

Here's the shape of my training and testing data:

    print(X_train.shape,Y_train.shape) 
    # Output: (693, 4) (693,)

    print(X_test.shape,Y_test.shape) 
    # Output: (149, 4) (149,)

The process I followed to build the Neural Network is:

     # define the keras model
     model = Sequential()
     model.add(Dense(4, input_dim=4,activation='relu'))
     model.add(Dense(4 ,activation='relu'))
     model.add(Dense(1,activation='sigmoid'))

     # compile the keras model
     model.compile(loss='binary_crossentropy',optimizer='adam', metrics=['accuracy'])

     # fit the keras model on the dataset
     hist = model.fit(X_train, Y_train, validation_data=(X_test, Y_test) ,epochs=10, batch_size=2)

The output after running hist :

Epoch 1/20
693/693 [==============================] - 1s 2ms/step - loss: 0.5974 - acc: 0.7605 - val_loss: 0.5499 - val_acc: 0.7785
Epoch 2/20
693/693 [==============================] - 0s 659us/step - loss: 0.5369 - acc: 0.7778 - val_loss: 0.5380 - val_acc: 0.7785
Epoch 3/20
693/693 [==============================] - 0s 700us/step - loss: 0.5330 - acc: 0.7778 - val_loss: 0.5369 - val_acc: 0.7785
Epoch 4/20
693/693 [==============================] - 0s 670us/step - loss: 0.5316 - acc: 0.7778 - val_loss: 0.5355 - val_acc: 0.7785
Epoch 5/20
693/693 [==============================] - 0s 720us/step - loss: 0.5307 - acc: 0.7778 - val_loss: 0.5345 - val_acc: 0.7785
Epoch 6/20
693/693 [==============================] - 0s 668us/step - loss: 0.5300 - acc: 0.7778 - val_loss: 0.5339 - val_acc: 0.7785
Epoch 7/20

Now, I would like to calcuate the precision , recall and F1-score instead of just the accuracy. I've tried following this . But I keep getting the following error:

ValueError: Classification metrics can't handle a mix of binary and continuous targets

Is there another way?

Answer 1

Can you try the following:

import numpy as np
from keras.callbacks import Callback
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score


class Metrics(Callback):

    def on_train_begin(self, logs={}):
        self.val_f1s = []
        self.val_recalls = []
        self.val_precisions = []

    def on_epoch_end(self, epoch, logs={}):
        val_predict = (np.asarray(self.model.predict(
            self.model.validation_data[0]))).round()
        val_targ = self.model.validation_data[1]
        _val_f1 = f1_score(val_targ, val_predict)
        _val_recall = recall_score(val_targ, val_predict)
        _val_precision = precision_score(val_targ, val_predict)
        self.val_f1s.append(_val_f1)
        self.val_recalls.append(_val_recall)
        self.val_precisions.append(_val_precision)
        print(f" — val_f1: {_val_f1} — val_precision: {_val_precision} — val_recall _val_recall")
        return


metrics = Metrics()

And in your code you need to use it as following:

hist = model.fit(X_train, Y_train,
                 validation_data=(X_test, Y_test), epochs=10,
                 batch_size=2, callbacks=[metrics])