Can the optimal learning rate differ for different architectures?

Question

In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this lecture (minute 59:55), the lecturer says that the learning rate is the first hyper-parameter that he tunes.

However, is it possible that the optimal learning rate is different for different architectures (for example, a different number of layers and neurons)? Or maybe the LR is architecture-independent and it depends only on the characteristics of the particular dataset we train our model on?

Moreover, should the LR be searched in the same process (e.g. grid-search) as the other hyper-parameters?

Answer 1

Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway.

The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one more training data point, the optimal learning rate will change. If you add one more neuron, it will change.

What you tune first is up to you, but once you have settled on an architecture, you should tune the learning rate first because it will have the largest effect on other hyper-parameters (normally, not always). For example tuning the number of epochs to train for first will be worthless if you start changing the learning rate because the network will learn at a different speed. That's why the learning rate is usually tuned first (after the architecture).