Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
As many papers point out, for better learning curve of a NN, it is better for a data-set to be normalized in a way such that values match a Gaussian curve.
Does this process of feature normalization apply only if we use sigmoid function as squashing function? If not what deviation is best for the tanh squashing function?
It has little to do with activation functions.
Say you have a 2 input Neural Net with 1 node in the hidden layer and 1 node in the output layer all with the sigmoid activation function.
Say suppose we are solving a bipolar classification problem. Now say one of the inputs is in the order of 10^4 and the other in order of 10 only. Now the Neural Net will propagate this values to the output layer via the hidden layer.
You get an error delta which is propagated back to the input layer.
Now as per gradient descent rule (if you look at the formula) the weight reduction will directly be proportional to delta * x_i where x_i is the ith input. SInce delta the net error due to both inputs is same for both, so The NN first decreases/increases the weight of a connection to make it to scale, and then after that the real learning which we are interested in occurs. Also if you see institutionally at the beginning if we give random weights the input which is larger will have more contribution to the output. It basically dictates the output. Now as the NN learns it reduces the weight-age of this large input to counter balance its high value. But if you do this at the beginning voila! the NN will train much faster.
Its like you have 2 kids, one naughtier than other. You leave them alone in a home and one of them breaks something (delta). But since you have no idea who did it you will contribute the delta to them equally. But as you learn about the kids nature your view (weight-age) about who broke things when you are not at home changes. Normalization is basically someone warning you already who the naughtier kid is and so you are able to take a balanced viewpoint from the beginning. (Very Bad example, but I could not think any better).
This example takes only 2 input so it might seem not much of a gain but in real life it will be if there are a large number of inputs and a lot of magnitude difference in them.
I may have missed something or other mathematical subtleties and I will be grateful if someone points them out.
Yes it applies, and no it shouldn't matter that much between the two activation functions.
