Questions tagged [tanh]
Use this tag for questions related to "hyperbolic tangent activation functions" (tanh) used in neural networks.
6 questions
9
votes
1
answer
2k
views
When to use Tanh?
When and why would you not use Tanh?
I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.
While using ReLU it reached 90% acc in >1000 training ...
- 125
3
votes
3
answers
6k
views
Why is there tanh(x)*sigmoid(x) in a LSTM cell?
CONTEXT
I was wondering why there are sigmoid and tanh activation functions in an LSTM cell.
My intuition was based on the flow of tanh(x)*sigmoid(x)
and the ...
- 242
3
votes
1
answer
1k
views
Why is tanh a "smoothly" differentiable function?
The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
The following excerpt taken from p4 of Neural Networks and Neural Language Models says that ...
- 3,990
2
votes
1
answer
283
views
Why and when do we use ReLU over tanh activation function?
I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how ...
1
vote
0
answers
140
views
Could we add clipping in the output layer of the actor in DDPG?
I have a doubt about how clipping affects the training of the RL agents.
In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:
...
- 69
0
votes
0
answers
26
views
Any LMs that use tanh (generalization) instead of sigmoid within Attention?
Question is in the title. Posts such as this and this mention how this would be possible. I have some colleagues who have anecdotally heard of this being done on very small transformer models but I ...
- 11