The tanh functions within the cell represent cell output or cell state. These are the values that either get passed on to other layers, or within the layer to the next time step. In theory, other activation functions could be used here according to taste, similar to other feed-forward or RNN networks. However, the -1 to 1 output range of tanh is useful, and I expect tanh has been experimentally validated as a good general case activation function here.

The sigmoid functions are used as soft gates for manipulating the raw RNN values. Importantly for your analysis, there is no sigmoid that takes the same input as any tanh. Each of the green boxes in the cell diagram in your question has a separate learnable set of weights applied to the combined input+last_state vector.

That means that your analysis of tanh(x)*sigmoid(x) is moot. The function is effectively tanh(x)*sigmoid(y) because inputs to each activation function can be radically different.

The intuition is that the LSTM can learn relatively "hard" switches to classify when the sigmoid function should be 0 or 1 (depending on the gate function and input data). As the weights are independent on the gates and input value processing components, the gradients to the cell output and state components are not composed in a combined function, but simply multiplied by the current value of the relevant switch. A muliplying hard switch of 1 will allow the gradient to flow back directly from the output loss to the point at which the gate decision was made - depending on which gate was activated, this improved gradient signal will either be to the input processing weights or the cell state procesing weights.

It is also possible for the input and cell state processing to be mixed in various combinations, and the gradient is not guaranteed strong. However, in situations requiring strong memory-like signals (such as using punctuation characters when processing text), it is possible to observe LSTM learning those signals, effectively classifying inputs with high confidence (close to either 0 or 1), thus creating toggle switches, counters etc, within the cell state vector.

The tanh functions within the cell represent cell output or cell state. These are the values that either get passed on to other layers, or within the layer to the next time step. In theory, other activation functions could be used here according to taste, similar to other feed-forward or RNN networks. However, the -1 to 1 output range of tanh is useful, and I expect tanh has been experimentally validated as a good general case activation function here.

The sigmoid functions are used as soft gates for manipulating the raw RNN values. Importantly for your analysis, there is no sigmoid that takes the same input as any tanh. Each of the green boxes in the cell diagram in your question has a separate learnable set of weights applied to the combined input+hidden_state vector.

That means that your analysis of tanh(x)*sigmoid(x) is moot. The function is effectively tanh(x)*sigmoid(y) because inputs to each activation function can be radically different.

The intuition is that the LSTM can learn relatively "hard" switches to classify when the sigmoid function should be 0 or 1 (depending on the gate function and input data). As the weights are independent on the gates and input value processing components, the gradients to the cell output and state components are not composed in a combined function, but simply multiplied by the current value of the relevant switch. A muliplying hard switch of 1 will allow the gradient to flow back directly from the output loss to the point at which the gate decision was made - depending on which gate was activated, this improved gradient signal will either be to the input processing weights or the hidden state procesing weights.

It is also possible for the input and cell state processing to be mixed in various combinations, and the gradient is not guaranteed strong. However, in situations requiring strong memory-like signals (such as using punctuation characters when processing text), it is possible to observe LSTM learning those signals, effectively classifying inputs with high confidence (close to either 0 or 1), thus creating toggle switches, counters etc, within the cell state vector.