Questions tagged [momentum]

For questions about the concept of momentum in the context of gradient descent.

Filter by
Sorted by
Tagged with
0 votes
0 answers
39 views

Is Momentum a form of inexact line search?

Is Momentum a form of inexact line search? I just want to know whether Momentum is considered a subset of inexact line search. By, inexact I mean that the step size is not the perfect one that ...
GaryTheBaddy's user avatar
1 vote
2 answers
70 views

At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
hanugm's user avatar
  • 3,581
3 votes
1 answer
1k views

How are these equations of SGD with momentum equivalent?

I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...
CuCaRot's user avatar
  • 892
1 vote
0 answers
72 views

Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs, where ...
SpiderRico's user avatar
9 votes
1 answer
1k views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
Dee's user avatar
  • 1,283
3 votes
3 answers
2k views

Why must the momentum factor be in the range 0-1?

Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
Ameba kupiec's user avatar