# What's the difference between hyperbolic tangent and sigmoid neurons?

Two common activation functions used in deep learning are the hyperbolic tangent function and the sigmoid activation function. I understand that the hyperbolic tangent is just a rescaling and translation of the sigmoid function:

$$\tanh(z) = 2\sigma(z) - 1$$.

Is there a significant difference between these two activation functions, and in particular, when is one preferable to the other?

I realize that in some cases (like when estimating probabilities) outputs in the range of $$[0,1]$$ are more convenient than outputs that range from $$[-1,1]$$. I want to know if there are differences other than convenience which distinguish the two activation functions.

Sigmoid > Hyperbolic tangent:

As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.

Hyperbolic tangent > Sigmoid:

Hyperbolic tangent has a property called "approximates identity near the origin" which means $$\tanh(0) = 0$$, $$\tanh'(0) = 1$$, and $$\tanh'(z)$$ is continuous around $$z=0$$ (as opposed to $$\sigma(0)=0.5$$ and $$\sigma'(0)=0.25$$). This feature (which also exists in many other activation functions such as identity, arctan, and sinusoid) lets the network learn efficiently even when its weights are initialized with small values. In other cases (eg Sigmoid and ReLU) these small initial values can be problematic.