# How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM?

So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.

TL;DR Here is a beautiful explanation with diagrams: source

the cell state is essentially long term memory embedding (correct me if I'm wrong)

The embedding can be long or short term and it is a vector.

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

Excerpt from source:

The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”

Formally the forget vector is: $$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$

So we see that it is actually a linear operation followed by a non-liner operation which restricts the values to be between 0 and 1. This is followed by an element wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing: $$C_{t-1}\odot f_t$$

Where $$\odot$$ is element wise multiplication.

Thus we see that:

The forget gate really is acting like a gate: It either lets values pass through or it pushes them toward zero. So it's purpose is to assign what values to forget/remember.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

This concatenation takes into account the "rolling" hidden state and the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.

In summary, the forget gate has the primary purpose of forming a vector of values between zero. This vector results by considering the current input and previous hidden state. The vector is used to forget/remember parts of the previous cell state via element wise multiplication. So no, it is not a black box it is a highly considerate non-linear operation through time.

Recall that multiplying a matrix on the right by a column vector produces a vector which is a linear combination of the columns of the matrix. So each column of $$W_f$$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $$W_f$$ form a sort of "basis" for a "toggle space."

When we concatenate to form $$[h_{t-1},x_t]$$ and then multiply we see that the term $$W_f\cdot[h_{t-1},x_t]$$ gives consideration to the hidden state and the input. That is, both $$h_{t-1}$$ and $$x_t$$ have there hand in toggling the gate.

The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example. That is, $$W_f\cdot[h_{t-1},x_t]$$ will be a vector in the column space of $$W_f$$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $$W_f$$. This "toggle" is then transformed into a "gate" in the "gate space."

So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $$[h_{t-1},x_t]$$. Formally:

$$f_t:[h_{t-1},x_t]\mapsto g_t$$

That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.

Important Note

The LSTM is only one of many types of forget-gate mappings. The main takeaway is that the LSTM works empirically for many applications.

That is a forget gate is an instance of a general "gate mapping":

$$f_t:v_t \mapsto g_t$$

Where $$v_t$$ could be any number of vectors resulting from any number of concatenations.

Conclusion

Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. One implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.

Edit: More Mathematical Intuition

Here is another view of what is happening in the LSTM from a dynamical point of view:

Video

• This answer makes sense from a neuron architecture standpoint. It just seems more about finding the right activation with the forget vector and then multiplying it with the cell state to get potentially useful activations for classification as the network recurs. This isn't quite the mathematically intuitive answer I was hoping for, this just seems to root back to the idea that the neural net is trying to emulate how neurons in the brain work. Just the theory of activation applied to what to forget over an embedding of memory is the TLDR I got. – user8714896 Dec 26 '19 at 11:25
• @user8714896 I think you mistake machine learning for a purely mathematical field. It's important to know that some models and concepts aren't developed through rigorous proofs of their effectiveness, and deterministic mathematics, but rather conceptually intuitive derivations. Take for example dropout, you can see how easy it would be to apply this to a network and see the benefits it provides, even if you don't have the mathematics to back it (however I do believe for dropout there actually is a mathematical intuition as well). – Recessive Dec 27 '19 at 0:55
• @user8714896 This is largely the result of the fact that machine learning is not really deterministic in regards to the methods one should use to achieve a result. No current field of math can tell you "To get the best convergence on a dataset, you should use this specific model with this activation function and combination of gates", and as such, a lot of that is left to the intuition of the human, rather then a purely mathematical derivation – Recessive Dec 27 '19 at 0:58
• @user8714896 Ah So you were looking for something like a demonstration of how the forget gate is somehow a mathematical consequence that can be derived in some manner. I believe Recessive and you are correct in this: the LSTM was more so a scientific result from experimentation and human intuition and not so much an exact mathematical result. I hope that the theory progresses enough to see how these architectures can "fall out" as natural consequences to the formulation of the theory. – respectful Dec 27 '19 at 14:11
• @user8714896 I found another resource analyzing the LSTM. I added the link in an edit – respectful Jan 10 at 22:31