Trained keras model much slower making its predictions than in training

Question

I trained a keras model overnight, and got 75% accuracy which I am happy with right now. It has 60,000 samples, each with a sequence length of 700, and a vocabulary of 30. Each epoch takes about 10 minutes on my gpu. So that's 60,000 / 600 seconds which is roughly 100 samples per second, and that has to include back propagation. So I saved my hdf5 file and loaded it again.

<code>#Model:
model = Sequential() 
model.add(LSTM(128, input_shape=(X.shape[1], X.shap[2]), return_sequences=True)) model.add(Dropout(0.25)) model.add(LSTM(64)) model.add(Dropout(0.25)) model.add(Dense(y.shape[1], activation='softmax'))
</code>

When I then make my predictions it is taking more like 1 second per prediction which is 100 times slower than training. The predictions are good, I've looked at small batches and I can use them. The problem is that I need many 100,000s of them. 10ms second per prediction would work, 1 second won't.

Can anyone suggest ways of speeding up Keras predictions?

Answer 1

I think it's because Keras's default predict behavior is with batch size 32. As a result especially if you're using a GPU, the small batch sizes destroy the performance. If you just change the batch size to predict(X_test, batch_size=128) you'll get significantly faster performance.