In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this lecture (minute 59:55), the lecturer says that the learning rate is the first hyper-parameter that he tunes.

However, is it possible that the optimal learning rate is different for different architectures (for example, a different number of layers and neurons)? Or maybe the LR is architecture-independent and it depends only on the characteristics of the particular dataset we train our model on?

Moreover, should the LR be searched in the same process (e.g. grid-search) as the other hyper-parameters?


Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway.

The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one more training data point, the optimal learning rate will change. If you add one more neuron, it will change.

What you tune first is up to you, but once you have settled on an architecture, you should tune the learning rate first because it will have the largest effect on other hyper-parameters (normally, not always). For example tuning the number of epochs to train for first will be worthless if you start changing the learning rate because the network will learn at a different speed. That's why the learning rate is usually tuned first (after the architecture).

  • $\begingroup$ so for example when I do grid search of optimal hyperparameters search, for every set of values I should additionally tune the learning rate? $\endgroup$
    – Makintosz
    Jul 29 at 12:46
  • $\begingroup$ Your explanation of why the optimal learning rate would change seems reasonable to me, but it may be a good idea to provide a link to some research paper or maybe an animation that shows some evidence for this. $\endgroup$
    – nbro
    Jul 30 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.