Why doesn’t this optimization method work?

Question

My idea is simple, for a loss function $L$, learning rate $\alpha$ and weights $W_n$ define the function $u_n(\alpha)$ as: $$u_n(\alpha) = L(W_n - \alpha \nabla_n L)$$ If we find the minimum of $u_n$, we’ve found the best learning rate for $\nabla_n L$. Using Newton-Raphson method to approximate the point in which $u_n’$ is zero we can find the minimum of $u_n$: $$\alpha_{n+1} = \alpha_n - \frac{u_n’}{u_n’’}$$ Finally update the weights: $$W_{n+1} = W_n - \alpha_{n+1} \nabla L$$

I used this method to train a neural network on MNIST(calculated $u_n’$ and $u_n’’$ using the finite difference method), and while it was better than regular SGD, it wasn’t impressive compared to Adam. Why doesn’t this method work well? Is there any way to improve it?

Answer 1

Your attempt to find an optimal adaptive learning rate at each update step makes sense, however, even the finite difference Newton-Ralphson application for your one-dimensional problem is likely sensitive and unstable numerically since the loss landscape in deep learning is nonconvex (far from quadratic as the classic Newton-Ralphson assumes) even locally. This can lead to learning rate updates unstable. You might try out more stable quasi-Newton methods such as BFGS, but they share the same sensitivity issue for second-order derivatives.

Also if you combine your method with mini-batch like most other deep learning stochastic optimization methods, they're incompatible since the deterministic result of Newton-Ralphson in one mini-batch may not generalize well to the next batch, leading to inconsistent update.