Keras 中的批量标准化的 output 是否取决于时期数？

Question

I am finding output of batchnormalization in Keras. My model is:

#Import libraries

import numpy as np
import keras
from keras import layers
from keras.layers import Input, Dense, Activation,  BatchNormalization, Flatten, Conv2D
from keras.models import Model

#Model

def HappyModel3(input_shape):


    X_input = Input(input_shape, name='input_layer')
    X = BatchNormalization(axis = 1, name = 'batchnorm_layer')(X_input)
    
    X = Dense(1, activation='sigmoid', name='sigmoid_layer')(X)
    
    
    model = Model(inputs = X_input, outputs = X, name='HappyModel3')
    
    return model

    

Compiling Model | here number of epochs is 1

X_train=np.array([[1,1,-1],[2,1,1]])
Y_train=np.array([0,1])

happyModel_1=HappyModel3(X_train[0].shape)
happyModel_1.compile(optimizer=keras.optimizers.RMSprop(), loss=keras.losses.mean_squared_error)
happyModel_1.fit(x = X_train, y = Y_train, epochs = 1 , batch_size = 2, verbose=0 )

finding Batch Normalisation layer's output for model with epochs=1:

for i in range(0, len(happyModel_1.layers)):
    
    tmp_model = Model(happyModel_1.layers[0].input, happyModel_1.layers[i].output)
    tmp_output = tmp_model.predict(X_train)
    
    if i in (0,1) :
        print(happyModel_1.layers[i].name)
        print(tmp_output.shape)
        print(tmp_output)
        print('\n')

Code Output is:

input_layer
(2, 3)
[[ 1.  1. -1.]
 [ 2.  1.  1.]]


batchnorm_layer
(2, 3)
[[ 0.99003249  0.99388224 -0.99551398]
 [ 1.99647105  0.99388224  0.9971655 ]]

We've normalized at axis=1 |Batch Norm Layer Output: At axis=1, 1st dimension mean is 1.5, 2nd dimension mean is 1, 3rd dimension mean is 0. Since its batch norm, I expect mean to be close to 0 for all 3 dimensions

This happens when I increase epochs to 1000:

happyModel_2=HappyModel3(X_train[0].shape)
happyModel_2.compile(optimizer=keras.optimizers.RMSprop(), loss=keras.losses.mean_squared_error)
happyModel_2.fit(x = X_train, y = Y_train, epochs = 1000 , batch_size = 2, verbose=0 )

finding Batch Normalisation layer's output for model with epochs=1000:

for i in range(0, len(happyModel_2.layers)):
    tmp_model = Model(happyModel_2.layers[0].input, happyModel_2.layers[i].output)
    tmp_output = tmp_model.predict(X_train)
    
    if i in (0,1) :
        print(happyModel_2.layers[i].name)
        print(tmp_output.shape)
        print(tmp_output)
        print('\n')

#Code output

input_layer
(2, 3)
[[ 1.  1. -1.]
 [ 2.  1.  1.]]


batchnorm_layer
(2, 3)
[[ -1.95576239e+00   8.08715820e-04  -1.86621261e+00]
 [  1.95795488e+00   8.08715820e-04   1.86590290e+00]]

We've normalized at axis=1 |Now At axis=1, batch norm layer output is: 1st dimension mean is 0, 2nd dimension mean is 0, 3rd dimension mean is 0. THIS IS AN EXPECTED OUTPUT NOW

My question is: Is output of Batch Normalization in Keras dependent on number of epochs? (Probably YES, as we do backpropagation, batch Normalization parameters will be affected by increasing number of epochs)

Answer 1

The keras documentation for BatchNormalization gives an answer to your question:

Importantly, batch normalization works differently during training and during inference.

What happens during training , ie when calling model.fit() ?

During training [...], the layer normalizes its output using the mean and standard deviation of the current batch of inputs.

But what will happen during inference, ie when calling mode.predict() as in your examples?

During inference [...], the layer normalizes its output using a moving average of the mean and standard deviation of the batches it has seen during training. That is to say, it returns (batch - self.moving_mean) / (self.moving_var + epsilon) * gamma + beta .

self.moving_mean and self.moving_var are non-trainable variables that are updated each time the layer in called in training mode [...].

It's important to understand that batch normalization will calculate the statistics (mean and variance) of your whole training data during training by looking at statistics of single batches and internally updating the moving_mean and moving_variance parameters by a running average computed form the single batch statistics. Therefore they're not affected by backpropagation. Ideally, after your model has seen enough training examples (or did enough training epochs), moving_mean and moving_variance will correspond to the statistics of your whole training set. These two parameters are then used during inference to normalize test examples. At the start of training the two parameters will be initialized to 0 and 1. Further batch norm has two more parameters called gamma and beta, which will be updated by the optimizer and therefore depend on your loss.

In essence, yes , the output of batch normalization during inference is dependent on the number of epochs you have trained your model. Firstly, due to changing moving averages for mean and variance and second due to learned parameters gamma and beta.

For a deeper understanding of how batch normalization works and why it is needed, have a look at the original publication .