2
$\begingroup$

Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class:

Rprop is equivalent to using the gradient, but also dividing by the size of the gradient

I don't understand this statement. Dividing by the size of the gradient would to me sound like scaling the gradient to be a unit vector. My understanding of Rprop does nothing like that (i.e. it just starts with small step size and then scales each dimension separately based on adjacent signs of the gradient).

Can someone elaborate on the details?

$\endgroup$

1 Answer 1

1
$\begingroup$

"Rprop is equivalent to using the gradient" means Rprop fundamentally relies on information about the sign of the gradient of each weight to determine the direction of weight updates. Like traditional gradient-based methods, Rprop utilizes information about whether the gradient is positive or negative to guide the weight updates.

The second part of Hinton's statement "but also dividing by the size of the gradient" roughly draws only an analogy to the concept of adaptive adjustment of the step size based on the magnitude of the gradient change for each weight instead of the usual fixed step size. The net effect is similar to the idea of always normalizing the gradient to have a fixed unit magnitude for weight updates to be consistently controlled and stable which ideally has no or little overshooting or oscillating convergence stability issue if an optimal solution actually exists.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .