Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?

Question

It is known that if $\alpha$ is set to high, then the cost function of the model may not converge. However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually $\alpha$ value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?

Answer 1

Setting too high a learning rate will extend the time to get a good result.

In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.