7
$\begingroup$

I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: The need for small learning rates on large problems

enter image description here

In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?

$\endgroup$
0
5
$\begingroup$

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.

The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing. Your ideal range consists of the learning rate values where the loss was decreasing steeply. After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.

Here is a concrete example from one of my experiments:

Learning Rate Finder Loss Plot

In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that). Your experiment may require different search bounds, but you could always start with that and adjust things as needed. At first, the loss plot is flat, and then it begins to decrease, but is too gradual. At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search. For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.

For more information, I suggest reading this Keras Learning Rate Finder post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.

$\endgroup$
4
$\begingroup$

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms.

In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient descent algorithm is not comparable to the learning rate in a tabular reinforcement learning algorithm such as Q-learning. This means that at a particular 'best' does not exist considering the different concepts denoted with the term `learning rate' in different algorithms.

Additionally, the learning rate is typically considered a part of the learning algorithm. The no free lunch theorem of machine learning tells us that no particular learning algorithm performs best across tasks. Because the learning rate is part of the solution, no particular learning rate is 'best' across tasks either.

In practice, you should set the learning rate sufficiently low to not 'overshoot' the optimal solution which will be evidenced by oscillations in the error (no convergence). But you also should also set it high enough to obtain reasonable performance given the amount of available training time.

What learning rate gives you the right trade-off typically requires a combination of domain knowledge and experimentation on the training set.

$\endgroup$
3
  • 3
    $\begingroup$ Nice answer except the second last paragraph is a bit vague. The NFL part only holds if non convex problems are allowed, otherwise smaller learning rates will always find an optimal solution found by larger learning rate if the problem is convex (almost always). Hence the second last para needs more explanation. $\endgroup$
    – user9947
    Feb 22 at 13:22
  • 2
    $\begingroup$ "In general, different algorithms assign different meaning to the same word 'learning rate'." This is wrong for your example. The meaning of "learning rate" in Q-learning is very similar to the meaning of "learning rate" in GD (if not the same thing): just look at the pseudocode!! In fact, in deep Q-learning, they are exactly the same thing. Maybe you need to clarify that the ideal range for the LR in one case may be different than in another case. $\endgroup$
    – nbro
    Feb 22 at 14:48
  • 2
    $\begingroup$ Moreover, note that now the question focuses on neural networks trained with gradient descent. Moreover, this answer doesn't not answer the question at all. The question is "is there an ideal range of learning rate which always gives a good result almost in all problems?". The question is not "How do I set my ideal learning rate?" $\endgroup$
    – nbro
    Feb 22 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.