When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first travel in one direction, then turn around and travel the other way for a while, then turn around and travel back, and so forth. This is wasteful, and the problem is that the learning rate for that parameter was too high, making it overshoot the optimal point. We don't want parameters to oscillate as they are trained; instead, ideally, they should settle directly to their final values, like a critically damped spring.
Is there an optimizer that sets learning rates based on this concept? Rprop seems related, in that it reduces the learning rate whenever the gradient changes direction. The problem with Rprop is that it only detects oscillations of period 2. What if the oscillation is longer, e.g. the parameter is moving in a sine wave with a period of dozens or hundreds of time steps? Looking for an optimizer that can suppress oscillations of any period length.
Let's be specific. Say that $w$ is a parameter, receiving a sequence of gradient updates $g_0, g_1, g_2, ... $ . I am looking for an optimizer that would pass the following tests:
- If $g_t = sin(t) - w$, then $w$ should settle to the value 0.
- If $g_t = sin(t) + 100 - 100 cos(0.00001t) - w$, then $w$ should settle to the value 100.
- If $g_t = sin(t) - w$ for $0 < t < 1000000$, and $g_t = sin(t) + 100 - w$ for $1000000 \leq t$, then $w$ should at first settle to the value 0, and then not too long after time step $1000000$ it should settle to the value 100.
- If $g_t = sin(t) - w$ for $floor(t / 1000000)$ even, and $g_t = sin(t) + 100 - w$ for $floor(t / 1000000)$ odd, then $w$ should at first settle to the value 0, then not too long after time step $1000000$ it should settle to the value 100, and then not too long after step $2000000$ it should settle back to 0, but eventually after enough iterations it should settle to the value 50 and stop changing forever after.