batch-normalization | 易学教程

Tensorflow batch_norm does not work properly when testing (is_training=False)

阅读更多关于 Tensorflow batch_norm does not work properly when testing (is_training=False)

问题 I am training the following model: with slim.arg_scope(inception_arg_scope(is_training=True)): logits_v, endpoints_v = inception_v3(all_v, num_classes=25, is_training=True, dropout_keep_prob=0.8, spatial_squeeze=True, reuse=reuse_variables, scope='vis') logits_p, endpoints_p = inception_v3(all_p, num_classes=25, is_training=True, dropout_keep_prob=0.8, spatial_squeeze=True, reuse=reuse_variables, scope='pol') pol_features = endpoints_p['pol/features'] vis_features = endpoints_v['vis/features'

How to modify batch normalization layers (DeconvNet) to be able to run with caffe?

阅读更多关于 How to modify batch normalization layers (DeconvNet) to be able to run with caffe?

问题 I wanted to run the Deconvnet on my data, however it seemd it has been written for another version of caffe . Does anyone know how to change batch_params ? The one that is in Deconvnet layers { bottom: 'conv1_1' top: 'conv1_1' name: 'bn1_1' type: BN bn_param { scale_filler { type: 'constant' value: 1 } shift_filler { type: 'constant' value: 0.001 } bn_mode: INFERENCE } } And the one that Caffe provides for cifar10 example: layer { name: "bn1" type: "BatchNorm" bottom: "pool1" top: "bn1" batch

Keras BatchNormalization only works for constant batch dim when axis=0?

阅读更多关于 Keras BatchNormalization only works for constant batch dim when axis=0?

问题 The following code shows one way that works and the other that fails. The BatchNorm on axis=0 should not depend on the batchsize or if it does it should be explicitly stated as such in the docs. In [118]: tf.__version__ Out[118]: '2.0.0-beta1' class M(tf.keras.models.Model): import numpy as np import tensorflow as tf class M(tf.keras.Model): def __init__(self, axis): super().__init__() self.layer = tf.keras.layers.BatchNormalization(axis=axis, scale=False, center=True, input_shape=(6,)) def

Implementing batch normalization in tensorflow where the graph is run multiple times before updating batch norm moving averages

阅读更多关于 Implementing batch normalization in tensorflow where the graph is run multiple times before updating batch norm moving averages

问题 I have a feed forward network with batch normalization layers in it. the network has to be run multiple times before one backpropogation can be applied. It is used to compute: a. vanilla logits (vl) b. logits with noisy inputs (nl) difference between vl and nl is also part of the cost function I cant understand how and when to run tf.GraphKeys.UPDATE_OPS. If i run it typically by grouping it with optimizer i think the moving averages will be updated with mean and standard deviation of noisy

Keras BatchNormalization population parameters update while training in tensorflow

阅读更多关于 Keras BatchNormalization population parameters update while training in tensorflow

问题 I am using Keras 2.0.8 with Tensorflow 1.3.0 in Ubuntu 16.04 with Cuda 8.0 and cuDNN 6. I am using two BatchNormalization layers( keras layers ) in my model and training using tensorflow pipeline . I am facing two problems here - BatchNorm layer population parameters ( mean and variance ) are not being updated while training even after setting K.learning_phase to True . As a result, inference is failing completely. I need some advice on how to update these parameters between training steps

fine-tune with batch normalization in keras

阅读更多关于 fine-tune with batch normalization in keras

问题 I have trained a model successfully over 100000 samples, which performs well both in train set and test set. Then, I tried to fine-tune it over one particular sample (one of the 100000 samples) and use the trained weights as the initialization. But the result is a little strange and I believe it is caused by the batch normalization layer. Specifically, my code can be listed as follows: model = mymodel() model.load_weights('./pre_trained.h5') #start from history rate = model.evaluate(x, y)

BatchNormalization in Keras

阅读更多关于 BatchNormalization in Keras

问题 How do I update moving mean and moving variance in keras BatchNormalization? I found this in tensorflow documentation, but I don't know where to put train_op or how to work it with keras models: update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) with tf.control_dependencies(update_ops): train_op = optimizer.minimize( loss ) No posts I found say what to do with train_op and whether you can use it in model.compile . 回答1: You do not need to manually update the moving mean and variances if

Where to apply batch normalization on standard CNNs

阅读更多关于 Where to apply batch normalization on standard CNNs

I have the following architecture: Conv1 Relu1 Pooling1 Conv2 Relu2 Pooling3 FullyConnect1 FullyConnect2 My question is, where do I apply batch normalization? And what would be the best function to do this in TensorFlow? The original batch-norm paper prescribes using the batch-norm before ReLU activation. But there is evidence that it's probably better to use batchnorm after the activation. Here's a comment on Keras GitHub by Francois Chollet: ... I can guarantee that recent code written by Christian [Szegedy] applies relu before BN. It is still occasionally a topic of debate, though. To your

BatchNormalization in Keras

阅读更多关于 BatchNormalization in Keras

How do I update moving mean and moving variance in keras BatchNormalization? I found this in tensorflow documentation, but I don't know where to put train_op or how to work it with keras models: update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) with tf.control_dependencies(update_ops): train_op = optimizer.minimize( loss ) No posts I found say what to do with train_op and whether you can use it in model.compile . You do not need to manually update the moving mean and variances if you are using the BatchNormalization layer. Keras takes care of updating these parameters during training,

Set “training=False” of “tf.layers.batch_normalization” when training will get a better validation result

阅读更多关于 Set “training=False” of “tf.layers.batch_normalization” when training will get a better validation result

问题 I use TensorFlow to train DNN. I learned that Batch Normalization is very helpful for DNN , so I used it in DNN. I use "tf.layers.batch_normalization" and follow the instructions of the API document to build the network: when training , set its parameter " training=True ", and when validate , set " training=False ". And add tf.get_collection(tf.GraphKeys.UPDATE_OPS) . Here is my code: # -*- coding: utf-8 -*- import tensorflow as tf import numpy as np input_node_num=257*7 output_node_num=257