当我在训练时间设置 is_training=False 时，为什么 Tensorflow BN 层中的moving_mean和moving _variance会变成nan？

Question

In the training time, I want to keep BN layer unchange, so I pass is_training=False to：

tf.contrib.layers.batch_norm(tensor_go_next, decay=0.9, center=True, scale=True, epsilon=1e-9,
                                              updates_collections=tf.GraphKeys.UPDATE_OPS,
                                              is_training=False, scope=name_bn_scope)

and didn't put name_bn_scope/gamma:0 name_bn_scope/beta:0 to train var_list.

After training, gamma and beta are still the same, which is what I want exactly. But the moving_mean and moving _variance would become nan matrix after training, which lead to the 0.1% accuracy.

I don't understand why, dosen't is_taining=False force tensorflow to keep moving_mean and moving _variance unchanged? How can I fix and implement this?

BN layer has tortured me for a so long time, Please help me！

Answer 1

Aha, I figure it out: the code block as shown below should be commented.(which is used to force Tensorflow to chagne moving_mean/moving_variance in bn layer when train_op ran, Since I don't want to chagne them in training. then it should be removed.)

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    train_op = optimizer.minimize(loss, name = 'train_op', var_list = var_list_to_train)

I also learned that when trapped in bugs, maybe go outside to take a break is the best way to figure out how to locate bugs and then solve it, which is a little bit like tricks in deep-learning to get out from local minimum.