It has little to do with activation functions.
Say you have a 2 input Neural Net with 1 node in the hidden layer and 1 node in the output layer all with the sigmoid activation function.
Say suppose we are solving a bipolar classification problem. Now say one of the inputs is in the order of 10^4
and the other in order of 10
only. Now the Neural Net will propagate this values to the output layer via the hidden layer.
You get an error delta
which is propagated back to the input layer.
Now as per gradient descent rule (if you look at the formula) the weight reduction will directly be proportional to delta * x_i
where x_i
is the i
th input. SInce delta
the net error due to both inputs is same for both, so The NN first decreases/increases the weight of a connection to make it to scale, and then after that the real learning which we are interested in occurs. Also if you see institutionally at the beginning if we give random weights the input which is larger will have more contribution to the output. It basically dictates the output. Now as the NN learns it reduces the weight-age of this large input to counter balance its high value. But if you do this at the beginning voila! the NN will train much faster.
Its like you have 2 kids, one naughtier than other. You leave them alone in a home and one of them breaks something (delta
). But since you have no idea who did it you will contribute the delta
to them equally. But as you learn about the kids nature your view (weight-age) about who broke things when you are not at home changes. Normalization is basically someone warning you already who the naughtier kid is and so you are able to take a balanced viewpoint from the beginning. (Very Bad example, but I could not think any better).
This example takes only 2 input so it might seem not much of a gain but in real life it will be if there are a large number of inputs and a lot of magnitude difference in them.
I may have missed something or other mathematical subtleties and I will be grateful if someone points them out.