In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this lecture (minute 59:55), the lecturer says that the learning rate is the first hyper-parameter that he tunes.
However, is it possible that the optimal learning rate is different for different architectures (for example, a different number of layers and neurons)? Or maybe the LR is architecture-independent and it depends only on the characteristics of the particular dataset we train our model on?
Moreover, should the LR be searched in the same process (e.g. grid-search) as the other hyper-parameters?