使用NaN进行Tensorflow / Keras批量标准化

Question

In a time sequence classification task, I used np.nan as an indicator for missing values (ie, end of sequence). I can calculate the length of the sequence by my own and use tf.nn.dynamic_rnn to build the RNN layers (I know it is deprecated, it is just a POC).

Is there a way to use the out-of-the-box batch normalization to do the normalization excluding the NANs? In a similar way to np.nanmean ? Or do I need to implement it from scratch?

For example, the following code outputs np.nan s:

import tensorflow as tf
import numpy as np

inp = np.array([
    # Sequence 1
    [ 
      [0.0, 0.1],
      [1.0, 0.2],
      [2.0, 0.3],
      [np.nan, np.nan],
      [np.nan, np.nan]
    ],
    # Sequence 2
    [
      [1.0, 0.2],
      [2.0, 0.3],
      [3.0, 0.4],
      [4.0, 0.5],
      [5.0, 0.6]
    ]
])

in_training_mode = tf.placeholder(tf.bool)
tf_inp = tf.placeholder(tf.float32, shape=inp.shape)
tf_bn = tf.keras.layers.BatchNormalization(axis=2)(
                tf_inp, training=in_training_mode
        )

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(tf_bn, feed_dict={tf_inp : inp, in_training_mode: True}))

# Output:
# [[[nan nan]
#  [nan nan]
#  [nan nan]
#  [nan nan]
#  [nan nan]]
# [[nan nan]
#  [nan nan]
#  [nan nan]
#  [nan nan]
#  [nan nan]]]

While the following equivalent code, we get the desired result:

print((inp - np.nanmean(inp, axis=(0,1))) / np.nanstd(inp, axis=(0,1)))

# Output:
# [[[-1.44115338 -1.44115338]
#  [-0.80064077 -0.80064077]
#  [-0.16012815 -0.16012815]
#  [        nan         nan]
#  [        nan         nan]]
# [[-0.80064077 -0.80064077]
#  [-0.16012815 -0.16012815]
#  [ 0.48038446  0.48038446]
#  [ 1.12089708  1.12089708]
#  [ 1.76140969  1.76140969]]]

Answer 1

I'm not sure how good of an idea it is to use nan as a value in a neural network. The correct way would be to set it to any invalid value and mask those. You can read about the masking layer here .