How to keep a list of models saved during training (save best only) in Keras?

Question

I am using Modelcheckpoint feature to save my models based upon "save best only" criteria.

file_name = str(datetime.datetime.now()).split(' ')[0] + f'{model_name}'+ '_{epoch:02d}.hdf5'

checkpoint_main = ModelCheckpoint(filename, monitor='val_acc', verbose=2,
                                save_best_only=True, save_weights_only=False,
                                mode='auto', period=1)

Since I am using "save best only" it will save only certain epochs. I want to collect the paths of actually saved models and save them to a list that I can access at the end of training. This list will be piped to other operations.

I tried looking at the source code, but I didn't see any examples of "train_end" that returns a list, so I am not really sure how to return something at the end of training.

https://github.com/keras-team/keras/blob/master/keras/callbacks.py#L360

Answer 1

If you wnat to store all the paths to saved models for each epoch, you could use a Callback , because Callback is just a python object and can collect data.

For exmaple, it can store a paths to the models in a list:

import datetime

class SaveEveryEpoch(Callback):
    def __init__(self, model_name, *args, **kwargs):
        self.model_checkpoint_paths = []
        self.model_name = model_name
        super().__init__(*args, **kwargs)

    def on_epoch_end(self, epoch, logs):
        # I suppose here it's a Functional model
        print(logs['acc'])
        path_to_checkpoint = (
            str(datetime.datetime.now()).split(' ')[0] 
            + f'_{self.model_name}'
            + f'_{epoch:02d}.hdf5'
        )
        self.model.save(path_to_checkpoint)
        self.model_checkpoint_paths.append(path_to_checkpoint)

• __init__ initializes an empty list and stores a model basic name
• on_epoch_end saves the model at the end of each epoch; also it appends a model path to list of model paths

Example of usage

from keras.datasets import mnist
from keras.models import Model
from keras.layers import Dense, Input
import numpy as np

(X_tr, y_tr), (X_te, y_te) = mnist.load_data()
X_tr = (X_tr / 255.).reshape((60000, 784))
X_te = (X_te / 255.).reshape((10000, 784))


def binarize_labels(y):
    y_bin = np.zeros((len(y), len(np.unique(y)))) 
    y_bin[range(len(y)), y] = 1
    return y_bin

y_train_bin, y_test_bin = binarize_labels(y_tr), binarize_labels(y_te)

model = Sequential()
model.add(InputLayer((784,)))
model.add(Dense(784, activation='relu'))
model.add(Dense(256, activation='relu'))
model.add(Dense(10, activation='softmax'))

model = Model(inp, out)
model.compile(loss='categorical_crossentropy', optimizer='adam')

Run the model with checkpoints

checkpoints = SaveEveryEpoch('mnist_model')
history = model.fit(X_tr, y_train_bin, callbacks=[checkpoints], epochs=3)
# ... training progress ...
checkpoints.model_checkpoint_paths
Out:
['2019-02-06_mnist_model_00.hdf5',
 '2019-02-06_mnist_model_01.hdf5',
 '2019-02-06_mnist_model_02.hdf5']

ls output:

2019-02-06_mnist_model_00.hdf5
2019-02-06_mnist_model_01.hdf5
2019-02-06_mnist_model_02.hdf5

Variations

Modify on_epoch_end to create some collection which can be ordered by a loss , for example ( logs argument contains a dictionary with loss and a metric named acc if some metric was provided). So, you can select a model with minimal loss/metric value later:

class SaveEveryEpoch(Callback):
    def __init__(self, model_name, *args, **kwargs):
        self.model_checkpoints_with_loss = []
        self.model_name = model_name
        super().__init__(*args, **kwargs)

    def on_epoch_end(self, epoch, logs):
        # I suppose here it's a Functional model
        print(logs['acc'])
        path_to_checkpoint = (
            str(datetime.datetime.now()).split(' ')[0] 
            + f'_{self.model_name}'
            + f'_{epoch:02d}.hdf5'
        )
        self.model.save(path_to_checkpoint)
        self.model_checkpoints_with_loss.append((logs['loss'], path_to_checkpoint))

Also, you can overload a default callback, for example, ModelCheckpoint to save all the paths, not only the best model, but I guess it's unnecessary in this case.