AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.

I've recently learned that it is not as helpful for RNNs where plain SGD is preferred.

For example, Deep Learning by Goodfellow et. al says: " .. both of these approaches have been largely replaced by simply using SGD (even without momentum) " Sec 10.11, pg.401.

Author talks about LSTMs and "both of these approaches" refer to second order and first order+momentum methods,respectively, according to my understanding.

What causes this discrepancy?

  • $\begingroup$ The momentum does not always help on CNN either $\endgroup$ – mirror2image Mar 9 at 5:52
  • $\begingroup$ @mirror2image Sure, but I suppose it's less common to use it for RNNs than CNNs. $\endgroup$ – SpiderRico Mar 9 at 5:55
  • $\begingroup$ Where did you read that momentum is not helpful for RNNs and it is useful for training CNNs? $\endgroup$ – nbro Mar 9 at 20:43
  • $\begingroup$ well some nlp papers here and there. I noticed they use plain SGD rather than using it with momentum. $\endgroup$ – SpiderRico Mar 9 at 21:00
  • $\begingroup$ @SpiderRico And what made you conclude that SGD with momentum isn't possibly more appropriate? $\endgroup$ – nbro Mar 9 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.