8

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. Also, using bold letters to represent vectors. In that notation, the update rule for basic gradient descent would be written as: $$\mathbf{w} \leftarrow \mathbf{...


3

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately. Equation 1 Consider the equation from the Stanford slide: $$ v_{t}=\rho v_{t-1}+\nabla f(x_{t-1}) \\ x_{t}=x_{t-1}-\alpha v_{t}, $$ Let's evaluate the first few $v_t$ so that we can arrive at a closed form solution: $v_0 = 0 \\ v_1 = \rho v_0 + ...


1

If gradient descent is like walking down a slope, momentum would be the literal momentum of the agent traversing the hyperplane. Under that analogy then, momentum factor would be analogous to the friction coefficient, with 1 being max friction and 0 being no friction. You should be able to see why there can't be friction beyond that range: if friction = 1 ...


Only top voted, non community-wiki answers of a minimum length are eligible