While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They are all dot products of a weight matrix and their inputs, plus bias, followed by a learned gating activation. Now, the difference between computation depends on the weight matrices being different from each other, that is, those weight matrices are specifically for specializing in the particular tasks like forgetting/keeping etc.

But these matrices are all initialized randomly, and it seems that there's no special tricks in the training scheme to make sure these weight matrices are learned in a manner that the associated gates specialize in their desired tasks. They are all random matrices that kept getting updated with gradient descent.

So how does, for example, a forget gate learn to function as a forgetting unit? Same question applies to others as well. Am I missing a part of the training for these networks? Can we ever say that these units learn truly disentangled functions from each other?


It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer.

Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (say, like a full stop). This forget gate, while it might not force a forget, has the ability to force one, and will likely learn to.

The input gate ultimately adds to the cell state. This gate doesn't have direct influence over the cell state (it can't make it 0, like the forget gate can), but it can add to it and influence it that way. So it is an input gate.

The output gate is used to interpret the hidden state, and get it ready to be combined with the cell state for a final output at that time step.

While these gates all use sigmoid functions, are all initialised randomly and have the same dimensionality, what their output is used in, and the order they're computed in gives them a certain role to play. Initially, they won't conform to this role, but logically as they learn, they likely will.

| improve this answer | |
  • $\begingroup$ There really isn't a completely clear reason why you need a hidden state over the cellstate. IIRC the structure of the LSTM allows for better gradient propagation(supposedly). $\endgroup$ – FourierFlux Jun 5 at 7:43
  • $\begingroup$ It does, the cell state acts as a gradient highway as looking through a series of LSTM cells, the gradient is only affected by 3 operations each time step, and so allows for longer distance relationships to remain in tact. $\endgroup$ – Recessive Jun 11 at 4:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.