# How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM?

So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), but I'm also assuming it's a matrix. Then the forget vector is calculated by concatenating the previous hidden state and the current input and adding the bias to it, then putting that through a sigmoid function that outputs a vector then that gets multiplied by the cell state matrix.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

I'd really like to understand the theory behind calculating the cell states and hidden states. Most people just tell me to treat it like a black box, but I think that, in order to have a successful application of LSTMs to a problem, I need to know what's going on under the hood. If anyone has any resources that are good for learning the theory behind why cell state and hidden state calculation extract key features in short and long term memory I'd love to read it.

TL;DR Here is a beautiful explanation with diagrams: source.

the cell state is essentially long term memory embedding (correct me if I'm wrong)

The embedding can be long or short term and it is a vector.

Why is the previous hidden state, current input and the bias put into a sigmoid function? Is there some special characteristic of a sigmoid that creates a vector of important embeddings?

Excerpt from source:

The sigmoid layer outputs numbers between zero and one, describing how much of each component should be let through. A value of zero means “let nothing through,” while a value of one means “let everything through!”

Formally the forget vector is:

$$f_t = \sigma(W_f\cdot[h_{t-1},x_t],+b_f)$$

So we see that it is actually a linear operation followed by a non-linear operation, which restricts the values to be between 0 and 1. This is followed by an element-wise operation on the previous cell state. That is we "gate"/"filter" the previous cell state doing:

$$C_{t-1}\odot f_t,$$

where $$\odot$$ is element-wise multiplication.

Thus we see that the forget gate really is acting like a gate: It either lets values pass through or it pushes them toward zero. So, it's purpose is to assign what values to forget/remember.

How does a concatenation of the hidden state of the previous input and the current input with the bias help with what to forget?

This concatenation takes into account the "rolling" hidden state and the current input. That is, the linear operation results in a new vector which can be seen as a "consideration embedding" of the current input and a "summary" of past inputs and cell states. The sigmoid then converts this "consideration embedding" into the forget vector.

In summary, the forget gate has the primary purpose of forming a vector of values between zero. This vector results by considering the current input and the previous hidden state. The vector is used to forget/remember parts of the previous cell state via element-wise multiplication. So, no, it is not a black box, it is a highly considerate non-linear operation through time.

Recall that multiplying a matrix on the right by a column vector produces a vector, which is a linear combination of the columns of the matrix. So each column of $$W_f$$ can be seen as a column of "toggles" that push up to 1 or pull down to 0 (once we apply sigmoid). That is, the columns of $$W_f$$ form a sort of "basis" for a "toggle space."

When we concatenate to form $$[h_{t-1},x_t]$$ and then multiply we see that the term $$W_f\cdot[h_{t-1},x_t]$$ gives consideration to the hidden state and the input. That is, both $$h_{t-1}$$ and $$x_t$$ have there hand in toggling the gate.

The LSTM has learned from the data how to make this toggling "meaningful." Below is a simple pictographic example.

That is, $$W_f\cdot[h_{t-1},x_t]$$ will be a vector in the column space of $$W_f$$ (ie the "toggle space"). Thus, the concatenated vector will "point" to the best "toggle" in the column ("toggle") space of $$W_f$$. This "toggle" is then transformed into a "gate" in the "gate space."

So intuitively, the range of the forget component is a sort of "gate space" where the domain is all vectors of form $$[h_{t-1},x_t]$$. Formally:

$$f_t:[h_{t-1},x_t]\mapsto g_t$$

That is to say that the forget gate learns the best mapping from the domain of concatenated hidden/input's to the range of possible gates.

Important Note

The LSTM is only one of many types of forget-gate mappings. The main takeaway is that the LSTM works empirically for many applications.

That is a forget gate is an instance of a general "gate mapping":

$$f_t:v_t \mapsto g_t,$$

where $$v_t$$ could be any number of vectors resulting from any number of concatenations.

Conclusion

Some data from the past is irrelevant to the current time step. We need a way to correctly "forget" irrelevant information. One way to "forget" is to use a forget gate mapping (like the one in LSTMs). Then we need to optimize the parameters of that forget gate. One implementation is used in the common LSTM architecture. Finally, the concatenation of the previous hidden state provides contextual information to the current input which is very helpful with forgetting.

More Mathematical Intuition

Here is another view of what is happening in the LSTM from a dynamic point of view: video.

• Comments are not for extended discussion; this conversation has been moved to chat. – nbro Jan 15 at 0:14