Tensorflow Multi-GPU丢失

Question

I am studying how to implement multi-GPU training on Tensorflow. Now I am reading this source as recommended in the documentation. As far as I understand, at line 178 variable loss accounts the loss for only one GPU (as the comment states). Thus, at the end of the cycle, say line 192, loss will retain the value of the loss of the last GPU considered. Variable loss is not modified till its use at line 243, when it is passed to Session.run() to be computed. So the loss value printed at line 255 is only the loss of the last GPU, and not the total one. It seems hard to me that Google engineers got wrong such a simple thing, what am I missing? Thanks!

Answer 1

It doesn't seem that you are missing something. They consider that printing the value of the loss and reporting the summaries for one tower is sufficient.

Generally you track the loss/summaries for each GPUs and/or compute the mean loss only for debugging when you start using a new model on multiple GPUs. Afterwards, only tracking one tower is sufficient since every tower contains the same copy of the model.

BTW, I find it way easier to use tf.estimators to do multi GPU training, using both tf.contrib.estimator.replicate_model_fn(...) and tf.contrib.estimator.TowerOptimizer(...) to distribute the model and optimizer.