4
$\begingroup$

I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever.

However, is this really the case? Are there instances where regular momentum performs better than Nesterov momentum or is Nesterov momentum performs at least as good as the regular momentum all the time?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that

Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence.

I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage does too - or at least, that it isn't applicable to typical neural network landscapes.

Perhaps it can be implemented more efficiently, but it seems to me that you need to do parameter updates twice (in order to calculate the gradient in a place you aren't moving two) and thus Nesterov requires more computation and memory than plain ole momentum.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I don't think momentum is added to improve convergence. Rather it's used to stabilize the NN (from oscillating) and help it in steep losses (ridges) and saddle points, as far as I know. $\endgroup$
    – user9947
    Feb 5, 2020 at 8:31
  • $\begingroup$ It depends what you mean by "improving" convergence - finding a better optimum or getting there faster. I think you mean the former (and are mostly correct, but there may be exceptions which are probably hard to quantify), while Goodfellow et al mean the latter (the following discussion discusses rate of convergence). $\endgroup$
    – Ketil Malde
    Feb 6, 2020 at 12:41
  • $\begingroup$ It's frustrating that the book makes this claim without any citation. $\endgroup$
    – japreiss
    Jun 1, 2022 at 22:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .