Feature Normalization/Standard Scalar in Keras

Question

I am working with a Sequential Keras model and I trying to figure out the best method for feature scaling.

model = Sequential()
model.add(Masking(mask_value=-50, input_shape=(None,10)))
model.add(LayerNormalization(axis=-1))
model.add(LSTM(100, input_shape=(None,10)))
model.add(Dense(100, activation='relu'))
model.add(Dense(3, activation='softmax'))
print(model.summary()) 

In line 3, I have a LayerNormalization layer which according to documentation, scales to mean and standard deviation. However, I have also come across Batch normalization and tf.keras.layers.experimental.preprocessing.Normalization . My question is is this method similar to Sklearn's StandardScalar() or is there another method I could use to feature scale within the model?

Answer 1

This should work. It uses an UpSampling layer for a naive 5x5 image-based input:

# define model
model = Sequential()
# define input shape, output enough activations for for 128 5x5 image
model.add(Dense(128 * 5 * 5, input_dim=100))
# reshape vector of activations into 128 feature maps with 5x5
model.add(Reshape((5, 5, 128)))
# double input from 128 5x5 to 1 10x10 feature map
model.add(UpSampling2D())
# fill in detail in the upsampled feature maps and output a single image
model.add(Conv2D(1, (3,3), padding='same'))
# summarize model
model.summary()

But you can use the Conv2DTranspose layer too, which combines the UpSampling2D and Conv2D layers into one layer.

A TimeDistributed layer in the case of LSTMs will help. Refer