[英]How to implement batch normalization layer for tensorflow multi-GPU code
I have created a multi-GPU network as Cifar10_multigpu我创建了一个多 GPU 网络作为Cifar10_multigpu
In the inference implementation, they said:在推理实现中,他们说:
We instantiate all variables using tf.get_variable() instead of tf.Variable() in order to share variables across multiple GPU training runs.
我们使用 tf.get_variable() 而不是 tf.Variable() 实例化所有变量,以便在多个 GPU 训练运行之间共享变量。 If we only ran this model on a single GPU, we could simplify this function by replacing all instances of tf.get_variable() with tf.Variable().
如果我们只在单个 GPU 上运行这个模型,我们可以通过用 tf.Variable() 替换 tf.get_variable() 的所有实例来简化这个函数。
So I did it about all my conv2d layers as the example, but what about batchnorm layer?所以我以我所有的 conv2d 层为例,但是 batchnorm 层呢? How do I implement it by myself?
我如何自己实现它?
Can I use tensorflow.contrib.slim.batch_norm
in this case?在这种情况下,我可以使用
tensorflow.contrib.slim.batch_norm
吗? The example does not contain any recommendation about batch norm layer.该示例不包含有关批处理规范层的任何建议。
Simply use tf.layers.batch_normalization
.只需使用
tf.layers.batch_normalization
。 It also creates variables via tf.get_variable()
, hence they can be shared as well.它还通过
tf.get_variable()
创建变量,因此它们也可以共享。
In addition, it works seamlessly with tf.layers.conv*
functions.此外,它还可以与
tf.layers.conv*
函数无缝tf.layers.conv*
。
Update : tf.nn.batch_normalization
is fine too.更新:
tf.nn.batch_normalization
也很好。 It's a more low-level function that requires you manage mean
and variance
tensors yourself.这是一个更底层的函数,需要您自己管理
mean
和variance
张量。 In fact, tf.layers.batch_normalization
is a wrapper over tf.nn.*
functions, which also includes tf.nn.fused_batch_norm
(a faster fused version).事实上,
tf.layers.batch_normalization
是tf.nn.*
函数的封装,其中还包括tf.nn.fused_batch_norm
(一个更快的融合版本)。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.