Hello I'm learning optimizers now, I can understand the momentum part (similar to physics world), but confused about different learning rate of different parameters,

for Adagrad/Rmsprop, if $$∂L/∂w_1$$ is large, then learning rate for $$w_1$$ is small, if ∂L/∂w_1 is small, then learning rate for $$w_1$$ is large. But mathematically, the -gradient is the steepest direction of value decreasing, for Adagrad/Rmsprop, it essential changes this direction to other direction, essentially changes the update direction more towards those partial derivative is small (if $$∂L/∂w_1$$ is small)

Why is that? My explain is, since Adagrad/Rmsprop essentially changes the update direction more towards those partial derivative is small,say $$w_1$$(if $$∂L/∂w_1$$ is small), that equals after take a step at -gradient direction, then take an extra step at $$w_1$$ direction since $$w_1$$ direction is flatter so it's less risk to take an extra step at $$w_1$$?

You can see that $$v^t$$ is a (corrected) exponential moving average of the second moment of the gradient