I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and hence, the network will not learn from far inputs. This is fine, however, the side effect of this problem is making the RNN lose the ability to memorize long sequences (memorizing far inputs). I don't understand what is the relationship between not learning the far inputs and not memorizing the far inputs.
In other words, how does not learning far inputs make the RNN forget far inputs?