0
$\begingroup$

I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and hence, the network will not learn from far inputs. This is fine, however, the side effect of this problem is making the RNN lose the ability to memorize long sequences (memorizing far inputs). I don't understand what is the relationship between not learning the far inputs and not memorizing the far inputs.

In other words, how does not learning far inputs make the RNN forget far inputs?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It is actually very simple, it is not about forgetting, but not learning at all means no memorization at all.

For long term dependencies, since the gradient is very small, what is learned is very little, so nothing was memorized either.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .