1
$\begingroup$

Hello I'm learning optimizers now, I can understand the momentum part (similar to physics world), but confused about different learning rate of different parameters,

for Adagrad/Rmsprop, if $∂L/∂w_1$ is large, then learning rate for $w_1$ is small, if ∂L/∂w_1 is small, then learning rate for $w_1$ is large. But mathematically, the -gradient is the steepest direction of value decreasing, for Adagrad/Rmsprop, it essential changes this direction to other direction, essentially changes the update direction more towards those partial derivative is small (if $∂L/∂w_1$ is small)

Why is that? My explain is, since Adagrad/Rmsprop essentially changes the update direction more towards those partial derivative is small,say $w_1$(if $∂L/∂w_1$ is small), that equals after take a step at -gradient direction, then take an extra step at $w_1$ direction since $w_1$ direction is flatter so it's less risk to take an extra step at $w_1$?

$\endgroup$

1 Answer 1

1
$\begingroup$

as reported in the original paper:

enter image description here

You can see that $v^t$ is a (corrected) exponential moving average of the second moment of the gradient

This doesn't mean that the gradient is penalized if it has entries very big, but that if during time, there are some entries that are constantly big, then they should be "penalized" (stepsize-wise)

Moreover, consider that being the second moment, it discards the sign of it. Indeed, this is usually done to reduce zig-zagging behaviors, because the "zipzag" will cause in some directions big gradients with opposite signs (but the second moment will discard such sign, only caring about the magnitude, and thus penalizing such directions)

enter image description here

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .