There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs.

Both of these sources mention the reason LSTMs are able to reduce the likelihood of the vanishing gradient problem is because

  1. The gradient contains the forget gate's vector of activions
  2. The addition of four gradient values help balance gradient values

I understand (1), but I don't understand what (2) means.

Any insight would greatly be appreciated!

  • $\begingroup$ can you please mention the slide number, where the point 2 is mentioned? $\endgroup$ – Swakshar Deb Oct 16 '20 at 19:00
  • $\begingroup$ 119. It's mentioned in the original post. $\endgroup$ – TNT Oct 16 '20 at 19:43
  • $\begingroup$ In order to vanish the gradient, the value should be zero for every path in the LSTM. If any path have the nonzero gradient the gradient will not be vanished. I think thats what they are trying to say. $\endgroup$ – Swakshar Deb Oct 16 '20 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.