[英]Keras BatchNormalization, What exactly is sample wise normalization?
I am trying to figure out what exactly the batch normalization in Keras does. 我想弄清楚Keras的批量标准化到底是什么。 Right now I have the following code. 现在我有以下代码。
for i in range(8):
c = Convolution2D(128, 3, 3, border_mode = 'same', init = 'he_normal')(c)
c = LeakyReLU()(c)
c = Convolution2D(128, 3, 3, border_mode = 'same', init = 'he_normal')(c)
c = LeakyReLU()(c)
c = Convolution2D(128, 3, 3, border_mode = 'same', init = 'he_normal')(c)
c = LeakyReLU()(c)
c = merge([c, x], mode = 'sum')
c = BatchNormalization(mode = 1)(c)
x = c
I set the batch norm mode to 1 which according to the Keras documentation 1: sample-wise normalization. This mode assumes a 2D input.
我将批量标准模式设置为1,根据Keras文档1: sample-wise normalization. This mode assumes a 2D input.
1: sample-wise normalization. This mode assumes a 2D input.
What I think this should be doing is just normalizing each sample in the batch independently of every other sample. 我认为这应该做的只是将批次中的每个样本标准化,而不是每个其他样本。 However when I look at the source code for the call function I see the following. 但是,当我查看调用函数的源代码时,我看到以下内容。
elif self.mode == 1:
# sample-wise normalization
m = K.mean(x, axis=-1, keepdims=True)
std = K.std(x, axis=-1, keepdims=True)
x_normed = (x - m) / (std + self.epsilon)
out = self.gamma * x_normed + self.beta
In this it is just computing the mean over all of x
which in my case is (BATCH_SIZE, 128, 56, 56)
I think. 在这里,它只计算所有x
的平均值,在我看来是(BATCH_SIZE, 128, 56, 56)
。 I thought it was supposed to normalize independent of the other samples in the batch when in mode 1. So shouldn't axis = 1
? 在模式1中,我认为它应该独立于批处理中的其他样本进行标准化。所以不应该axis = 1
? Also what does "assumes a 2D input" mean in the documentation? 在文档中还有什么“假设2D输入”?
In this it is just computing the mean over all of x which in my case is
(BATCH_SIZE, 128, 56, 56)
I think. 在这里,它只计算所有x的平均值,在我看来是(BATCH_SIZE, 128, 56, 56)
。
By doing this you are already violating the contract of that layer. 通过这样做,您已经违反了该图层的合同。 That is not a 2 dimensional but a 4 dimensional input. 这不是二维而是四维输入。
I thought it was supposed to normalize independent of the other samples in the batch when in mode 1 在模式1中,我认为它应该独立于批处理中的其他样本进行标准化
It does. 确实如此。 K.mean(..., axis=-1)
is reducing axis -1 which is synonymous for the last axis of the input. K.mean(..., axis=-1)
正在减小轴-1,它与输入的最后一个轴同义。 So assuming an input shape of (batchsz, features)
, axis -1 will be the features
axis. 因此,假设输入形状为(batchsz, features)
,轴-1将成为features
轴。
Since K.mean
is very similar to numpy.mean
you can test this yourself: 由于K.mean
是非常相似的numpy.mean
您可以测试这个自己:
>>> x = [[1,2,3],[4,5,6]]
>>> x
array([[1, 2, 3],
[4, 5, 6]])
>>> np.mean(x, axis=-1)
array([ 2., 5.])
You can see that the features got reduced per sample in the batch. 您可以看到批次中每个样本的特征都减少了。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.