Questions tagged [momentum]

For questions about the concept of momentum in the context of gradient descent.

Filter by
Sorted by
Tagged with
1
vote
1answer
137 views

How are these equations of SGD with momentum equivalent?

I know this question may be so silly, but I can not prove it. In Stanford slide (page 17), they define the formula of SGD with momentum like this: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{...
1
vote
0answers
64 views

Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs, where ...
8
votes
1answer
781 views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...