I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: The need for small learning rates on large problems
In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?
The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing. Your ideal range consists of the learning rate values where the loss was decreasing steeply. After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.
Here is a concrete example from one of my experiments:
In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that). Your experiment may require different search bounds, but you could always start with that and adjust things as needed. At first, the loss plot is flat, and then it begins to decrease, but is too gradual. At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search. For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.
For more information, I suggest reading this Keras Learning Rate Finder post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.
Edit: I recently learned that in newer versions of Tensorflow / Keras, the on_train_batch_end callback now returns an aggregated average loss instead of the raw loss for the given batch, which gives poor results for the learning rate finder. See this Github Issue for more information and the workaround I am currently using.
The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms.
In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient descent algorithm is not comparable to the learning rate in a tabular reinforcement learning algorithm such as Q-learning. This means that at a particular 'best' does not exist considering the different concepts denoted with the term `learning rate' in different algorithms.
Additionally, the learning rate is typically considered a part of the learning algorithm. The no free lunch theorem of machine learning tells us that no particular learning algorithm performs best across tasks. Because the learning rate is part of the solution, no particular learning rate is 'best' across tasks either.
In practice, you should set the learning rate sufficiently low to not 'overshoot' the optimal solution which will be evidenced by oscillations in the error (no convergence). But you also should also set it high enough to obtain reasonable performance given the amount of available training time.
What learning rate gives you the right trade-off typically requires a combination of domain knowledge and experimentation on the training set.
