1
$\begingroup$

It is known that if $\alpha$ is set to high, then the cost function of the model may not converge. enter image description here However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually $\alpha$ value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?

$\endgroup$
1
$\begingroup$

Setting too high a learning rate will extend the time to get a good result.

In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.