Keras没有使用完整的CPU内核进行培训

Question

I am training a LSTM model on a very huge dataset on my machine using Keras on Tensorflow backend. My machine have 16 cores. While training the model I noticed that the load in all the cores are below 40%.

I have gone through different sources looking for a solution and have tried providing the cores to use in the backend as

config = tf.ConfigProto(device_count={"CPU": 16})
backend.tensorflow_backend.set_session(tf.Session(config=config))

Even after that the load is still the same.

Is this because the model is very small.? It is taking around 5 minutes for an epoch. If it uses full cores the speed can be improved.

How to tell Keras or Tensorflow to use the full available cores ie 16 cores to train the model.??

I have went through these stackoverflow questions and tried the solutions mentioned there. It didn't help.

Limit number of cores used in Keras

Answer 1

How are you training the model exactly? You might want to look into using model.fit_generator() but with a Keras Sequence object instead of a custom generator. This allows to safely use multiprocessing and will result in all cores being used.

You can checkout the Keras docs for an example.