多 GPU tensorflow 代码中的批量标准化参数更新？

Question

I have written a multi gpu Cnn code

In this link they commented in line 249

# Retain the Batch Normalization updates operations only from the # final tower. Ideally, we should grab the updates from all towers # but these stats accumulate extremely fast so we can ignore the # other stats from the other towers without significant detriment.

but this code of updating batch norm(in line 253) :

with tf.device('/gpu:%d' % i):
.
.
.
  batchnorm_updates = 
  tf.get_collection(slim.ops.UPDATE_OPS_COLLECTION,scope)

is done for all towers(GPUs) so what is the right place of this update?

Answer 1

I think you understand the code incorrectly.

as the for loop in the code ,

for i in range(FLAGS.num_gpus):
  with tf.device('/gpu:%d' % i):
    with tf.name_scope('%s_%d' % (inception.TOWER_NAME, i)) as scope:

      with slim.arg_scope([slim.variables.variable], device='/cpu:0'):

      ......

      batchnorm_updates = tf.get_collection(slim.ops.UPDATE_OPS_COLLECTION,
                                            scope)

the batchnorm_updates will be replaced after each for loop , so it just contains the last tower's Batch Normalization updates operations when the for loop is done.