I came across this post https://paperswithcode.com/methods/category/learning-rate-schedules which lists some different learning rate schedules and the number of papers which use them. I was a bit surprised to see 'linear warmup with linear decay' and 'cosine annealing' so prominently on this list, because I had never heard of either. On the other hand, inverse square root decay, step decay, exponential decay (which I had heard of) don't have many papers associated with them.

Does this ranking hold any weight? Specifically, is linear warmup with linear decay considered to be the "best" learning rate schedule (i.e. the one which tends to work the best in practice)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.