In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/

Prof Recht shows two plots:

enter image description here

enter image description here

He says one of the reasons the plot below has a lower train-test gap is because that model was trained with a lower learning rate (and he also manually drops the learning rate at 120 epoch).

Why would a lower learning rate reduce overfitting?

  • $\begingroup$ bigger learning rate makes the loss curve oscillation high, and it works for training data only $\endgroup$ – datdinhquoc Jul 14 '20 at 9:59
  • 1
    $\begingroup$ BTW, two represented figures have not the same y-scale and it can be confusing and deceptive! $\endgroup$ – OmG Jul 14 '20 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.