1
$\begingroup$

I have started reading Fundamentals of Deep Learning by Nikhil Buduma and I have a question regarding tanh neurons. In the book, it is stated:

"When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."

Can anyone explain me why exactly??

$\endgroup$
1

1 Answer 1

1
$\begingroup$

It should be mentioned that RELU is the current activation function standard. But to answer your question: The importance here is that it is very common to have normalized your data (e.g. using batch normalization), then the data is centered around 0.

As @DuttaA commented, look at this answer from Cross-Validated:

Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].

And

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .