简体繁体 English

如何用python计算批量标准化？

[英]How to calculate batch normalization with python?

原文 2020-01-08 17:57:40 2 1 python/ tensorflow/ deep-learning/ batch-normalization

When I implement batch normalization in python from scrach, I am confused.当我从 scrach 在 python 中实现批量标准化时，我很困惑。 Please see A paper demonstrates some figures about normalization methods, I think it may be not correct.请看一篇论文展示了一些关于归一化方法的数字，我认为它可能不正确。 The description and figure are both not correct.描述和图都不正确。

Description from the paper:论文中的描述：

Figure from the paper:论文中的图： BN的原图 As far as I am concerned, the representation of batch normalization is not correct in the original paper.就我而言，原始论文中批量归一化的表示是不正确的。 I post the issue here for discussion.我把这个问题贴在这里供讨论。 I think the batch normalization should be like the following figure.我认为批量归一化应该如下图所示。

The key point is how to calculate mean and std.关键是如何计算均值和标准差。 With feature maps' shape as (batch_size, channel_number, width, height) , mean = X.mean(axis=(0, 2, 3), keepdims=True) or mean = X.mean(axis=(0, 1), keepdims=True)特征图的形状为(batch_size, channel_number, width, height) , mean = X.mean(axis=(0, 2, 3), keepdims=True) or mean = X.mean(axis=(0, 1), keepdims=True)

Which one is correct?哪一个是正确的？

1 个解决方案

You should calculate mean and std across all pixels in the images of the batch.您应该计算批次图像中所有像素的均值和标准差。 So use axis=(0, 2, 3) parameters.所以使用 axis=(0, 2, 3) 参数。 If the channels have roughly same distributions - you may calculate mean and std across channels as well.如果通道具有大致相同的分布 - 您也可以跨通道计算均值和标准差。 so just use mean() and std() without axes parameter.所以只需使用没有轴参数的 mean() 和 std() 。

The figure in the article is correct - it takes mean and std across H and W (image dimensions) for each batch.文章中的数字是正确的 - 每个批次的 H 和 W（图像尺寸）均采用均值和标准差。 Obviously, channel is not shown in the 3d cube.显然，通道未显示在 3d 立方体中。