Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Question

Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation. The goals of such work have been a balance of the goals of artificial network training in general.

• Minimal convergence time given a specific set of computing resources
• Maximal accuracy in convergence with regard to the training acceptance criteria
• Maximal reliability in achieving acceptable test results after training is complete

The development of a surface involving these three measurements would require multiple training experiments, but may provide a relationship that itself could be approximated either by curve fitting or by a distinct deep artificial network using the experimental results as examples.

• Epoch index
• Learning rate hyper-parameter value
• Observed rate of convergence

The goal of such work would be to develop, via manual application of analytic geometry experience or via deep network training the following function, where

• $\alpha$ is the ideal learning rate for any given epoch indexed by $i$,
• $\epsilon$ is the loss function result, and
• $\Psi$ is a function the result of which approximates the ideal learning rate for as large an array of learning scenarios possible within a clearly defined domain.

$\alpha_i = \Psi (\epsilon, i)$

The development of arriving at $\Psi$ as a closed form (formula) would be of general academic and industrial value.

Has this been done?

Answer 1

Has this been done?

Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), it has not been done to the level of suggesting a global function worth approximating.

The problem is that learning rate tuning, like other hyperparameter tuning, is highly dependent on the problem at hand, plus the other hyperparamater values currently in use, such as size of layers, which optimiser is in use, what regularisation is being used, activation functions.

Although you may be hoping for $\Psi(\epsilon, i | P)$ to exist where P is the problem domain, it likely does not except as a mean value over all $\Psi(\epsilon, i | D, H)$ for the problem domain, where D is the dataset and H all the other hyperparameters.

It is likely that such a function exists, of ideal learning rate for best expected convergence per epoch. However, it would be incredibly expensive to sample it with enough detail to make approximating it useful. Coupled with limited applicability (not domain-specific, but linked to data and other hyperparameters), a search through all possible learning rate trajectories looks like it would give poor return on investment.

Instead, the usual pragmatic approaches are:

• Include learning rate in hyperparameter searches, such as grid search, random search, genetic algorithms and other global optimisers.

• Decay learning rate using one of a few approaches that have been successfully guessed and experiments have shown working. These have typically been validated by plotting learning curves of loss functions or other metrics, and the same tracking is usually required in new experiments to check that the approach is still beneficial.

• Some optimisers use a dynamic learning rate parameter, which is similar to your idea but based on reacting to measurements during learning as opposed to changes based on an ideal function. They have a starting learning rate, then adjust it based on heuristics derived from measuring learning progress. These heuristics can be based on per-epoch measurements, such as whether a validation set result is improving or not. One such approach is to increase learning rate whilst results per epoch are improving, and reduce learning rate if results are not improving, or have got worse.

I have tried this last option, on a Kaggle competition, and it worked to some extent for me, but did not really improve results overall - I think it is one of many promising ideas in ML that can be made to work, but that has not stayed as a "must have", unlike say dropout, or using CNNs for images.

Some optimisers store a multiplier per layer or even per weight - RMSProp and Adam for example track rate of change of each parameter, and adjust the rate for each weight during updates. These can work very well in large networks, where the issue is not so much needing a specific learning rate at any time, but that a single learning rate is too crude to cover the large range of gradients and differences in gradients across the index space of all the connections. With RMSProp and Adam, the need to pick specific learning rates or explore them is much reduced, and often a library's default is fine.