I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever.

However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?


The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that

Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.

I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.

Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.

| improve this answer | |
  • $\begingroup$ I don't think momentum is added to improve convergence. Rather it's used to stabilize the NN (from oscillating) and help it in steep losses (ridges) and saddle points, as far as I know. $\endgroup$ – DuttaA Feb 5 at 8:31
  • $\begingroup$ It depends what you mean by "improving" convergence - finding a better optimum or getting there faster. I think you mean the former (and are mostly correct, but there may be exceptions which are probably hard to quantify), while Goodfellow et al mean the latter (the following discussion discusses rate of convergence). $\endgroup$ – Ketil Malde Feb 6 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.