BN层学习

uB=1m∑mi=1xi...(1)$$u_B=\frac{1}{m}{\mathrm{\sum }}_{i=1}^m{x}_i...(1)$$

σ2B=1m∑mi=1(xiuB)2...(2)$${\sigma }_B^2=\frac{1}{m}{\mathrm{\sum }}_{i=1}^m(x_i{u}_B{)}^2...(2)$$

xi=(xiuB)2σ2B+√...(3)$$x_i^{}=\frac{(x_i{u}_B{)}^2}{\sqrt{{\sigma }_B^2+}}...(3)$$

yi=γxi+β=BNγβ(xi)...(4)$$y_i=\gamma x_i^{}+\beta =B{N}_{\gamma \beta }(x_i)...(4)$$

γ$\gamma$和偏移系数β$\beta$,从而使网络去学习输出的特征分布.在实际测试中,我们只输入一个测试样本时,此时没有mini batch这个概念了,其他参数如均值和标准差都是固定不变的,即测试时均值和标准差的公式为:

E[x]=EB[uB]...(5)$$E[x]=E_B[u_B]...(5)$$

Var[x]=mm1EB[σ2B]...(6)$$Var[x]=\frac{m}{m1}{E}_B[{\sigma }_B^2]...(6)$$

y=γVar[x]+√x+(βγE[x]Var[x]+√)...(7)$$y=\frac{\gamma }{\sqrt{Var[x]+}}x+(\beta \frac{\gamma E[x]}{\sqrt{Var[x]+}})...(7)$$

def batch_normalization(x, mean, variance, offset, scale, variance_epsilon, name=None):

β$\beta$(默认为0),scale是缩放系数γ$\gamma$(默认为1),variance_epsilon为$$,调用这个api就可以输出BN后的结果.
另一个实现BN操作的api是tf.contrib.layers.batch_norm()函数,详情可见:https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm

参考:
1.https://blog.csdn.net/hjimce/article/details/50866313
2.https://zhuanlan.zhihu.com/p/26138673
3.https://morvanzhou.github.io/tutorials/machine-learning/ML-intro/3-08-batch-normalization/

文章来源: BN层学习