如何在 Keras 的每个时期保存训练历史？

Question

I can't keep my PC running all day long, and for this I need to save training history after every epoch. For example, I have trained my model for 100 epochs in one day, and on the next day, I want to train it for another 50 epochs. I need to generate the loss vs epoch and accuracy vs epoch graphs for the whole 150 epochs. I am using fit_generator method. Is there any way to save the training history after every epoch (most probably using Callback )? I know how to save the training history after the training has ended. I am using Tensorflow backend.

Answer 1

Keras has the CSVLogger callback which appears to do exactly what you need; from the documentation :

Callback that streams epoch results to a CSV file.

It has an append parameter for adding to the file. Again, from the documentation:

append : Boolean. True: append if file exists (useful for continuing training). False: overwrite existing file

from keras.callbacks import CSVLogger

csv_logger = CSVLogger("model_history_log.csv", append=True)
model.fit_generator(...,callbacks=[csv_logger])

Answer 2

I had a similar requirement, I went for a naive approach.

1.Python code to run for 50 Epochs:
I saved the history of the model and the model itself trained for 50 epochs. .history is used to store entire history of the trained model.

history = model.fit_generator(......) # training the model for 50 epochs
model.save("trainedmodel_50Epoch.h5") # saving the model
with open('trainHistoryOld', 'wb') as handle: # saving the history of the model
    dump(history.history, handle)

2.Python code for loading the trained model and training for another 50 epochs:

from keras.models import load_model
model = load_model('trainedmodel_50Epoch.h5')# loading model trained for 50 Epochs

hstry = model.fit_generator(......) # training the model for another 50 Epochs

model.save("trainedmodel_50Epoch.h5") # saving the model 

with open('trainHistoryOld', 'wb') as handle: # saving the history of the model trained for another 50 Epochs
    dump(hstry.history, handle)

from pickle import load
import matplotlib.pyplot as plt

with open('trainHistoryOld', 'rb') as handle: # loading old history 
    oldhstry = load(handle)

oldhstry['loss'].extend(hstry['loss'])
oldhstry['acc'].extend(hstry['acc'])
oldhstry['val_loss'].extend(hstry['val_loss'])
oldhstry['val_acc'].extend(hstry['val_acc'])

# Plotting the Accuracy vs Epoch Graph
plt.plot(oldhstry['acc'])
plt.plot(oldhstry['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

# Plotting the Loss vs Epoch Graphs
plt.plot(oldhstry['loss'])
plt.plot(oldhstry['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

You can create custom class too as mentioned in the answer provided earlier.

Answer 3

To save model history you have two options.

1. Use keras ModelCheckPoint callback class
2. Create custom class

Here is how to create custom checkpoint call back class.

class CustomModelCheckPoint(keras.callbacks.Callback):
    def __init__(self,**kargs):
        super(CustomModelCheckPoint,self).__init__(**kargs)
        self.epoch_accuracy = {} # loss at given epoch
        self.epoch_loss = {} # accuracy at given epoch

    def on_epoch_begin(self,epoch, logs={}):
        # Things done on beginning of epoch. 
        return

    def on_epoch_end(self, epoch, logs={}):
        # things done on end of the epoch
        self.epoch_accuracy[epoch] = logs.get("acc")
        self.epoch_loss[epoch] = logs.get("loss")
        self.model.save_weights("name-of-model-%d.h5" %epoch) # save the model

Now to use the call back class

checkpoint = CustomModelCheckPoint()
model.fit_generator(...,callbacks=[checkpoint])

now checkpoint.epoch_accuracy dictionary contains accuracies at a given epoch and checkpoint.epoch_loss dictionary contains losses at a given epoch

Answer 4

You can save the training history as follows

hist = model.fit_generator(generator(features, labels, batch_size), samples_epoch=50, nb_epoch=10)
import pickle
with open('text3', 'wb') as f:
    pickle.dump(hist.history, f)

To save the training history after each epoch

import pickle
hist1 = []
for _ in range(10):
   hist = model.fit(X_train, y_train, epochs=1, batch_size=batch_size, validation_split=0.1)
   hist1.append(hist.history)

with open('text3', 'wb') as f:
    pickle.dump(hist1.history, f)

For the checkpoint

filepath="weights.best.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
# Fit the model
model.fit(X, Y, validation_split=0.33, epochs=150, batch_size=10, callbacks=callbacks_list, verbose=0)