Batch normalization instead of input normalization

Question

Can I use batch normalization layer right after input layer and not normalize my data? May I expect to get similar effect/performance?

In keras functional it would be something like this:

x = Input (...)
x = Batchnorm(...)(x)
...

Answer 1

You can do it. But the nice thing about batchnorm, in addition to activation distribution stabilization, is that the mean and std deviation are likely migrate as the network learns.

Effectively, setting the batchnorm right after the input layer is a fancy data pre-processing step. It helps, sometimes a lot (e.g. in linear regression). But it's easier and more efficient to compute the mean and variance of the whole training sample once, than learn it per-batch. Note that batchnorm isn't free in terms of performance and you shouldn't abuse it.

---