I was wondering why there are sigmoid and tanh activation functions in an LSTM cell.
My intuition was based on the flow of
and the derivative of
It seems to me that authors wanted to choose such a combination of functions, the derivative would make possible big changes around the 0, since we can use normalized data and weights. Another thing is that the output would go to 1 for positive values and go to 0 for negative values which is convenient.
On the other hand, it seems natural that we use sigmoid in forget gate, since we want to have a better focus on the important data. I just don't understand why there cannot only be a sigmoid function in the input gate.
What I found on the web is this article where the author claims:
To overcome the vanishing gradient problem, we need a method whose second derivative can sustain >for a long range before going to zero. Tanh is a good function that has all the above properties.
However, he doesn't explain why this is the case.
Also, I found the opposite statement here, where the author says that the second derivative of the activation function should go to zero, however, there is no proof for that claim.
- Why cannot we put a signal with just a sigmoid on the input gate?
- Why there are
tanh(x)*sigmoid(x)signals in the input and output gate?