In Keras, what is the best between using validation_split (in “fit” method) and model.evaluate function?

Question

In Keras there are two ways (at least) to split the data and display loss/accuracy:

1. In Keras fit function there is the validation_split option that allows to split the dataset into training and testing sets AND to display loss/accuracy values during the training.

2. Another way to split the data is to split it at the begining of the code (using train_test_split function for instance), train the data with the training set THEN use model.evaluate to test the test data.

I realize that the accuracy with the second method is generally lower (maybe is it more realistic?) and I am wondering which method should be definitely prefered on the other , I would think that method 2) being less optimistic would be preferable.

You can play with my code here , to use the 1st method, uncomment lines where written "HERE1" (and comment lines with "HERE2") and to use the 2nd method do the opposite.

Any suggestion?

Best regards

Aymeric

Answer 1

The best way to see your validation results would be to split your training and validation data into equal amounts of each class. This can be done using StratifiedKFold from sklearn.model_selection .

When I looked through the docs for tf.keras.Model , I found that "The validation data is selected from the last samples in the x and y data provided, before shuffling". Looking further into the last 30% of your training data, I found that 156 out of the 186 samples contained the positive class, leaving just 30 samples as the negative class. In the data that you split afterwards, only 115 of the samples were positive out of the 268. Note that the difference in the total number of samples is due to splitting the training dataset in model.fit() rather than on the entire dataset before training. This large imbalance could be a contributor towards the large variation in accuracy.

Due to this, I would recommend almost never using the validation_split parameter because it is easier to organize your data beforehand than checking the last x% of your training data during training.