Keras培训陷入LSTM

Question

I'm trying to run a LSTM model in Keras but get stuck in the training part.

For each epoch, it takes around 3-4 seconds for the model to train the steps to 49x/500, then the model will get stuck. After like 7xx seconds the training will resume and complete the remaining few steps and finish ONE epoch.

Then it loops again trains very fast then freezes.

What is the possible reason?

The code I run is the coding example P.213 from the book Deep Learning with Python by Francois Chollet. If the code/my hardware have a problem, the training process for each epoch should be constantly slow? Now it trains very fast at the beginning but gets stuck at the end for each epoch.

I have tried update GPU driver, conda update --all, assign another GPU to run the model (I have 2 GPU).

I'm sure my GPU are fine because I have no problem running other models.

Answer 1

Ths is normal, at the end of each epoch Keras will use your validation data to compute validation loss and metrics, and this of course takes time, maybe somehow your validation set is bigger than your training set?

It looks like it freezes but it is indeed computing on the validation set, nothing to worry about.