My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long Short-Term Memory, model the keeping and forgetting mechanisms explicitly with sigmoid activations, namely gates. What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?


It is not the sigmoid in particular. LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget).

Observe the top horizontal line in the image below. The line represents the "internal state" $C$. Such state is not used as output for the outside world; it is only passed from the cell to itself at different time steps. This is where the old information is "stored".

The sigmoid is a convenient means to write information in $C$, as the output of a sigmoid ranges between $[0,1]$. For example: think of computing point-wise multiplication between $C$ and some other generic vector $v$. If all values of $v$ are in range $[0,1]$, we are essentially masking $C$, i.e. selectively erasing information from it.

LSTMs try to learn the values of such vector $v$. To make sure that is it in the desired range, sigmoid activations are used.

LSTM cell

For more information, I recommend consulting Goodfellow's book Deep Learning, that has a section dedicated to recurrent nets and sequence modeling in general.

  • $\begingroup$ Thanks for the answer. Pictured is the Gated Recurrent Unit, correct? So it is this unit in particular that is responsible for the keeping and forgetting of past information from different time scales, rather than just the sigmoid activation itself? $\endgroup$ Jun 5 at 14:08
  • 1
    $\begingroup$ The picture is not a Gated Recurrent Unit. It is a Long Short Term Memory (LSTM) unit, which is similar to a GRU, but not exactly the same. Apart from that, you are correct: it is not the sigmoid that is responsible for keeping and forgetting information, but rather the cell architecture as a whole. $\endgroup$
    – m-pana
    Jun 5 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.