Keras Batch Training回调串联

Question

I have a model in Keras, and due to a large dataset, I load it in batches using a generator. So the training routine is encapsulated in a for loop:

from keras.callbacks import History 

history = History()

for epoch in xrange(100):
  x_train,y_train=load_data_generator()
  model.fit(x_train, y_train, nb_epoch=1, batch_size=1000, verbose=2, validation_data=(x_test, y_test), callbacks=[history])

I would ideally like to have history for the entire training loop. However, keras seems to overwrite the history during each loop with the current batch.

Question 1) How can I easily concatenate history over all epochs? In other words trick it into resuming where it left off in the prior history.

Question 2) Each time keras completes an epoch, it prints the number of the epoch. Since I do training in batches, is there an easy way of updating the epoch? I've seen a 'initial_epoch' parameter for the fitting routine, but it sounds like that would revert to a prior epoch.

Answer 1

Can you try to set the initial_epoch = epoch inside the fit call?

I would assume that it will then append the information to your History callback since it's a new epoch every time. The number of the epoch will also be incremented at each step of the loop.

I hope this helps.

Answer 2

You may find the documentation on image preprocessing in Keras helpful in understanding the use of data generators. I believe that both of your issues will be solved by using the fit_generator() function instead of fit() .