I think a better way to understand LSTMs is by their purpose, instead of gradients and distributions.

If you analyze the interactions of each gate with the cell state, you'll realize that LSTMs essentially implement differentiable memory as a counter.

There are 2 important things to realize:

• Each sigmoid/tanh is preceded by a linear projection, so the output of the sigmoid of the forget gate is different to that of the input gate.
• Sigmoid outputs are mostly 0 or 1, while tanh outputs are mostly -1 or 1.

Following the numbering in the diagram:

1. The forget gate generates a binary mask which is multiplied with the previous cell state. Items with 0 in the mask are 'reset' to 0, while items with 1 in the mask are passed through unchanged.
2. The input gate generates 2 masks using sigmoid and tanh, whose product is added to the cell state. The tanh mask with, -1 or +1 outputs, determines whether to increment or decrement items in the cell state. The sigmoid mask determines whether an item should be updated at all, similarly to the forget gate.
3. The output gate determines what to expose to the subsequent layer and state using a similar logic. The tanh here acts as a 'binarizer' to simplify the cell state (which can theoretically be from -infinity to +infinity) to a binary -1 or +1 value, while the sigmoid masks out irrelevant items.

So while you could replace sigmoid and tanh with other activation functions, they make the most sense in this context and have the added benefit of well-defined derivatives. Remember that LSTM's were introduced in 1997 when tanh and sigmoid activation functions were still popular, and ReLU had not taken over yet (~2011).

I think a better way to understand LSTMs is by their purpose, instead of gradients and distributions.

If you analyze the interactions of each gate with the cell state, you'll realize that LSTMs essentially implement differentiable memory as a counter.

There are 2 important things to realize:

• Each sigmoid/tanh is preceded by a linear projection, so the output of the sigmoid of the forget gate is different to that of the input gate.
• Sigmoid outputs are mostly 0 or 1, while tanh outputs are mostly -1 or 1.

Following the numbering in the diagram:

1. The forget gate generates a binary mask which is multiplied with the previous cell state. Items with 0 in the mask are 'reset' to 0, while items with 1 in the mask are passed through unchanged.
2. The input gate generates 2 masks using sigmoid and tanh, whose product is added to the cell state. The tanh mask with, -1 or +1 outputs, determines whether to decrement or increment items in the cell state. The sigmoid mask determines whether an item should be updated at all, similarly to the forget gate.
3. The output gate determines what to expose to the subsequent layer and state using a similar logic. The tanh here acts as a 'binarizer' to simplify the cell state (which can theoretically be from -infinity to +infinity) to a binary -1 or +1 value, while the sigmoid masks out irrelevant items.

So while you could replace sigmoid and tanh with other activation functions, they make the most sense in this context and have the added benefit of well-defined derivatives. Remember that LSTM's were introduced in 1997 when tanh and sigmoid activation functions were still popular, and ReLU had not taken over yet (~2011).