Questions tagged [momentum]
For questions about the concept of momentum in the context of gradient descent.
3
questions
1
vote
1answer
137 views
How are these equations of SGD with momentum equivalent?
I know this question may be so silly, but I can not prove it.
In Stanford slide (page 17), they define the formula of SGD with momentum like this:
$$
v_{t}=\rho v_{t-1}+\nabla f(x_{t-1})
\\
x_{t}=x_{...
1
vote
0answers
64 views
Why do momentum techniques not work well for RNNs?
AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
I've recently learned that it is not as helpful for RNNs, where ...
8
votes
1answer
781 views
What is the formula for the momentum and Adam optimisers?
In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is:
$$w\ -= r \times g$$
where $r$ is the ...
