How do I know if my Neural Network model is overfitting or not (Keras)

Question

I'm using Keras to predict if I'll get an output of 1 or 0. The data looks like this:

    funded_amnt  emp_length  avg_cur_bal  num_actv_rev_tl    loan_status
    10000       5.60088      19266                 2                  1
    13750       5.60088      2802                  6                  0
    26100       10.0000      19241                17                  1

The target is loan_status and the features are the remaining. I've normalized the data before starting to build a Neural Network model.

Here's the shape of my training and testing data:

    print(X_train.shape,Y_train.shape) 
    # Output: (693, 4) (693,)

    print(X_test.shape,Y_test.shape) 
    # Output: (149, 4) (149,)

The process I followed to build the Neural Network is:

     # define the keras model
     model = Sequential()
     model.add(Dense(4, input_dim=4,activation='relu'))
     model.add(Dense(4 ,activation='relu'))
     model.add(Dense(1,activation='sigmoid'))

     # compile the keras model
     model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

     # fit the keras model on the dataset
     hist = model.fit(X_train, Y_train, epochs=10, batch_size=2)

The output after running hist :

         Epoch 1/10
         693/693 [==============================] - 2s 2ms/step - loss: 0.6379 - acc: 0.7013
         Epoch 2/10
         693/693 [==============================] - 0s 611us/step - loss: 0.5207 - acc: 0.7951
         Epoch 3/10
         693/693 [==============================] - 0s 605us/step - loss: 0.5126 - acc: 0.7951
         Epoch 4/10
         693/693 [==============================] - 0s 621us/step - loss: 0.5109 - acc: 0.7951
         Epoch 5/10
         693/693 [==============================] - 0s 611us/step - loss: 0.5105 - acc: 0.7951
         Epoch 6/10
         693/693 [==============================] - 0s 636us/step - loss: 0.5091 - acc: 0.7951
         Epoch 7/10
         693/693 [==============================] - 0s 644us/step - loss: 0.5090 - acc: 0.7951
         Epoch 8/10
         693/693 [==============================] - 0s 659us/step - loss: 0.5086 - acc: 0.7951
         Epoch 9/10
         693/693 [==============================] - 0s 668us/step - loss: 0.5083 - acc: 0.7951
         Epoch 10/10
         693/693 [==============================] - 0s 656us/step - loss: 0.5076 - acc: 0.7951

it's all almost the same and doesn't change after the second Epoch. I've tried changing number of Epochs and Batch size but keep getting the same results. Is this normal? or is it a sign of overfitting and I need to change some parameters

Answer 1

Your test data meant to be for monitoring the model's overfitting on train data :

hist = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=10, batch_size=2)

During the training you will reach a point, where the train loss continues to decrease, but your test loss stops to decrease. That the point where your data starts to overfit.

In statistics, overfitting is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably".

As an extreme example, if the number of parameters is the same as or greater than the number of observations, then a model can perfectly predict the training data simply by memorizing the data in its entirety. Such a model, though, will typically fail severely when making predictions.

Usually a learning algorithm is trained using some set of "training data ": exemplary situations for which the desired output is known. The goal is that the algorithm will also perform well on predicting the output when fed " validation data " that was not encountered during its training. Overfitting is especially likely in cases where learning was performed too long or where training examples are rare, causing the learner to adjust to very specific random features of the training data, that have no causal relation to the target function. In this process of overfitting, the performance on the training examples still increases while the performance on unseen data becomes worse .

The green line represents an overfitted model and the black line represents a regularized model. While the green line best follows the training data, it is too dependent on that data and it is likely to have a higher error rate on new unseen data, compared to the black line.

Answer 2

Overfitting is not your problem right now, it can appear in models with a high accurrancy (>95%), you should try training more your model. If you want to check if your model is suffering overffiting, try to forecast using the validation data. If the acurrancy looks too low and the training acurrancy is high, then it is overfitting, maybe.

Answer 3

If you are overfitting, your training loss will continue decreasing, but the validation accuracy doesn't improve. The problem in your case is that your network doesn't have enough capacity to fit the data, or the features you are using doesn't have enough information to perfectly predict the loan status.

You can solve this by either increasing the capacity of your network by adding some layers, dropout, regularization, etc. Or by adding more training data and more features if possible.