10
votes
Accepted
When to use Tanh?
Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive ...
- 425
4
votes
Accepted
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
The tanh functions within the cell represent cell output or cell state. These are the values that either get passed on to other layers, or within the layer to the next time step. In theory, other ...
- 33.3k
4
votes
Accepted
Why is tanh a "smoothly" differentiable function?
A smooth function is usually defined to be a function that is $n$-times continuously differentiable, which means that $f$, $f'$, $\dots$, $f^{(n - 1)}$ are all differentiable and $f^{(n)}$ is ...
- 1,010
3
votes
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
I think a better way to understand LSTMs is by their purpose, instead of gradients and distributions.
If you analyze the interactions of each gate with the cell state, you'll realize that LSTMs ...
- 31
2
votes
Why and when do we use ReLU over tanh activation function?
For a discussion about the advantages of ReLU, see the original paper by Glorot (2011) "Deep sparse rectifier neural networks".
"Efficient Backprop" is a 1998 paper. At the time ...
- 215
1
vote
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
The purpose of the tanh and sigmoid functions in an LSTM (Long Short-Term Memory) network is to control the flow of information through the cell state, which is the "memory" of the network.
...
- 111
Only top scored, non community-wiki answers of a minimum length are eligible