Tensorflow 2 with multiple GPUs

Question

I am trying to training an LSTM network on tensorflow2 using 8 GPUs. Here is the code I am using:

max_seq_length = 100
embedding_vector_length = 32
corpuse_count = len(word_count)
model = Sequential() 
model.add(Embedding(corpuse_count , embedding_vector_length, input_length=max_seq_length)) 
model.add(LSTM(64))
model.add(Dense(16, activation='sigmoid'))
parallel_model = multi_gpu_model(model, gpus=8)
parallel_model.compile(loss='categorical_crossentropy',optimizer='rmsprop', metrics=['accuracy'])
parallel_model.fit(X_train_t1_proc, y_train_proc, validation_split=0.15, epochs=3, batch_size=64) 

However when I check the actual GPU performance only one the gpus are working. Does anyone know how I can solve this?

Answer 1

There could be many reasons. First of all, if you're using tensorflow.keras, note that multi_gpu_model is deprecated and is/will be removed, and you better use tf.distributed.MirroredStrategy instead. The given link shows examples of how to properly use multi_gpu_model .

Another reason could be that your batch size is so small that it does not take full advantage of multiple gpus. Everytime you call nvidia-smi to check gpu usage, even though it has allocated maximum memory, it is not making any calculations at the moment.

To make sure only needed GPU memory is allocated, try dynamically growing the gpu memory as discussed here

I hope that helps