Why would moving_mean and moving _variance in Tensorflow BN layer become nan when I set is_training=False in training time?

问题

In the training time, I want to keep BN layer unchange, so I pass is_training=False to：

tf.contrib.layers.batch_norm(tensor_go_next, decay=0.9, center=True, scale=True, epsilon=1e-9,
                                              updates_collections=tf.GraphKeys.UPDATE_OPS,
                                              is_training=False, scope=name_bn_scope)

and didn't put name_bn_scope/gamma:0 name_bn_scope/beta:0 to train var_list.

After training, gamma and beta are still the same, which is what I want exactly. But the moving_mean and moving _variance would become nan matrix after training, which lead to the 0.1% accuracy.

I don't understand why, dosen't is_taining=False force tensorflow to keep moving_mean and moving _variance unchanged? How can I fix and implement this?

BN layer has tortured me for a so long time, Please help me！

回答1:

Aha, I figure it out: the code block as shown below should be commented!(which is used to force Tensorflow to chagne moving_mean/moving_variance in bn layer when train_op ran. Since I don't want to chagne them in training, then it should be removed.)

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    train_op = optimizer.minimize(loss, name = 'train_op', var_list = var_list_to_train)

I also learned that when trapped in bugs, maybe go outside to take a break is the best way to figure out how to locate bugs and then solve it, which is a little bit like tricks in deep-learning to get out from local minimum.

来源：https://stackoverflow.com/questions/62442306/why-would-moving-mean-and-moving-variance-in-tensorflow-bn-layer-become-nan-whe