Keras model not saving correctly

Question

I am training a neural network in Keras but when new data comes and I try to retrain it, the loss in the epochs it's as high as the first time I trained my model.

 checkpoint = ModelCheckpoint('my_model.h5', monitor='loss', verbose=1, save_best_only=True, mode='min')
 callbacks_list = [checkpoint]
 model.fit(X_train,y_train, batch_size = batch_size, epochs = epochs, callback = callbacks_list)
 new_model = load_model('my_model.h5')

As suggested here Keras: How to save model and continue training? I tried to predict the same data both in model and new_model and measure the differences with:

assert_allclose(model.predict(x_train),
            new_model.predict(x_train),
            1e-5)

In fact, I got the Assertion Error and I did even with tol = 1e-2 so that makes me think my model is not loading as it should. Anyone has a clue of why is this happening?

Answer 1

ModelCheckpoint saves the model weighs that has had less loss in training.

Your model has saved the last epoch weighs.

If the last epoch of your model is not the one that has had less loss, the weights of the saved model ( new_model ) do not match those of the original model and the prediction is not the same.