Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I have started reading Fundamentals of Deep Learning by Nikhil Buduma and I have a question regarding tanh neurons. In the book, it is stated:
"When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."
Can anyone explain me why exactly??
It should be mentioned that RELU is the current activation function standard. But to answer your question: The importance here is that it is very common to have normalized your data (e.g. using batch normalization), then the data is centered around 0.
As @DuttaA commented, look at this answer from Cross-Validated:
Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].
And
The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
