Normalizing to [0,1] vs [-1,1]

Question

I\'ve been going through a few tutorials on using neural networks for key points detection. I\'ve noticed that for the inputs (images) it\'s very common to divide by 255 (normal

Answer 1

According to me, technically there should not be much of a difference on how you are normalising the values.

But these things matter in ML techniques.

Normalising the pixel range from (0 to 255 ) to (0 to 1) makes the convergence rate faster. Here you can do ranging between -1 and 1 as well. I have used this range in lot of problems. And there are no as such issues.

But for the output it is little tricky. Using range 0 to 1 is not a better idea because of the activation function you are using. ReLU is max(0, x) which works better when you provide negative values as well. That is the whole point of relu. Also tanh ranges values between -1 and 1. The only choice you are left with is to use sigmoid function which does not perform that well compared to relu and tanh functions. The problem with sigmoid is of vanishing gradient and it is not zero centered which gives somewhat zig zagged gradient updates for the weights. You can look for it here.