Keras : How to save model weights while training within a single eopch?

Question

I am training a CNN in Keras. It will take around 18 hours for each epoch. I need to save the model weights every half an hour as I cannot run the program for straight 18 hours.

I tried saving the model weights using checkpoint with period = 0.0125 . I thought this will save the model weights at every 100 training examples (assuming the total no. of training examples = 8000)

cp_callback = tf.keras.callbacks.ModelCheckpoint(checkpoint_path, verbose=1, save_weights_only=True,period=0.0125)

model.fit_generator(
    training_set,
    steps_per_epoch=8400,
    epochs=25,
    callbacks = [cp_callback],
    validation_data=test_set,
    validation_steps=2165)

Answer 1

ModelCheckpoint saves model after every epoch. The period is the interval between the epochs not within the epoch.

One plausible solution is to write your own custom callback which saves the weights at every batch end or beginning.

class myCallback(keras.callbacks.Callback):
def on_batch_end(self, batch,logs={}):
    self.model.save_weights('D:/downloads/model_batch'+str(batch)+'.hdf5')

callbacks1 = myCallback()

You can apply any condition like when to save, what to save etc. depending on your problem. Hope this helps