batch-normalization

问题 What will happen when I use batch normalization but set batch_size = 1 ? Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Normally, I know, when batch_size = 1 , variance will be 0. And (x-mean)/variance will lead to error because of division by 0. But why did errors not occur when I set batch_size = 1 ? Why my network was trained as good as I expected? Could anyone explain it? Some people argued that: The

问题 What will happen when I use batch normalization but set batch_size = 1 ? Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Normally, I know, when batch_size = 1 , variance will be 0. And (x-mean)/variance will lead to error because of division by 0. But why did errors not occur when I set batch_size = 1 ? Why my network was trained as good as I expected? Could anyone explain it? Some people argued that: The

问题 How do I set weights of the batch normalization layer of Keras? I am bit confused by the documentation weights: Initialization weights. List of 2 Numpy arrays, with shapes: [(input_shape,), (input_shape,)] Note that the order of this list is [gamma, beta, mean, std] Do we need all four [gamma, beta, mean, std]? Is there a way to set weights using only [gamma, beta]? 回答1: Yes, you need all four values. Recollect what batch normalization does. Its goal is to normalize (i.e. mean = 0 and

问题 I am a little confused about how should I use/insert "BatchNorm" layer in my models. I see several different approaches, for instance: ResNets: "BatchNorm" + "Scale" (no parameter sharing) "BatchNorm" layer is followed immediately with "Scale" layer: layer { bottom: "res2a_branch1" top: "res2a_branch1" name: "bn2a_branch1" type: "BatchNorm" batch_norm_param { use_global_stats: true } } layer { bottom: "res2a_branch1" top: "res2a_branch1" name: "scale2a_branch1" type: "Scale" scale_param {

问题 I want to apply Layer Normalisation to recurrent neural network while using tf.compat.v1.nn.rnn_cell.LSTMCell . There is a LayerNormalization class but how should I apply this in LSTMCell. I am using tf.compat.v1.nn.rnn_cell.LSTMCell because I want to use projection layer. How should I achieve Normalisation in this case. class LM(tf.keras.Model): def __init__(self, hidden_size=2048, num_layers=2): super(LM, self).__init__() self.hidden_size = hidden_size self.num_layers = num_layers self.lstm

问题 TensorFlow seems to implement at least 3 versions of batch normalization: tf.nn.batch_normalization tf.layers.batch_normalization tf.contrib.layers.batch_norm These all have different arguments and documentation. What is the difference between these, and which one should I use? 回答1: They are actually very different. nn.batch_normalization performs the basic operation (i.e. a simple normalization) layers.batch_normalization is a batchnorm "layer", i.e. it takes care of setting up the trainable

问题 I have a network that trains the Batch Norm (BN) layer. My batch size is 16, hence, I must use multiple GPUs. I have followed the example of inceptionv3 that can be summarized as with tf.Graph().as_default(), tf.device('/cpu:0'): images_splits = tf.split(axis=0, num_or_size_splits=FLAGS.num_gpus, value=images) labels_splits = tf.split(axis=0, num_or_size_splits=FLAGS.num_gpus, value=labels) for i in range(FLAGS.num_gpus): with tf.device('/gpu:%d' % i): with tf.name_scope('%s_%d' % (inception

问题 I have the following architecture: Conv1 Relu1 Pooling1 Conv2 Relu2 Pooling3 FullyConnect1 FullyConnect2 My question is, where do I apply batch normalization? And what would be the best function to do this in TensorFlow? 回答1: The original batch-norm paper prescribes using the batch-norm before ReLU activation. But there is evidence that it's probably better to use batchnorm after the activation. Here's a comment on Keras GitHub by Francois Chollet: ... I can guarantee that recent code written

问题 Suppose we have a simple Keras model that uses BatchNormalization: model = tf.keras.Sequential([ tf.keras.layers.InputLayer(input_shape=(1,)), tf.keras.layers.BatchNormalization() ]) How to actually use it with GradientTape? The following doesn't seem to work as it doesn't update the moving averages? # model training... we want the output values to be close to 150 for i in range(1000): x = np.random.randint(100, 110, 10).astype(np.float32) with tf.GradientTape() as tape: y = model(np.expand

问题 I am trying to make a simple GANs to generate digits from the MNIST dataset. However when I get to training(which is custom) I get this annoying warning that I suspect is the cause of not training like I'm used to. Keep in mind this is all in tensorflow 2.0 using it's default eager execution. GET THE DATA(not that important) (train_images,train_labels),(test_images,test_labels) = tf.keras.datasets.mnist.load_data() train_images = train_images.reshape(train_images.shape[0], 28, 28, 1).astype(