# Questions tagged [momentum]

For questions about the concept of momentum in the context of gradient descent.

6 questions
Filter by
Sorted by
Tagged with
1 vote
27 views

### At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
919 views

### What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...
31 views

### Limit of momentum update equation

I am self-studying on optimization algorithm on https://d2l.ai/chapter_optimization/momentum.html and couldn't get my head around some derivation: Instead of the standard gradient descent update ...
2k views

### Why must the momentum factor be in the range 0-1?

Why is it a bad idea to have a momentum factor greater than 1? What are the mathematical motivations/reasons?
I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this:  v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...