Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate (using a tailored function)?

I know that the update of the parameters is done via $\gamma \nabla L$, where $\nabla L$ is the gradient and $\gamma$ the learning rate, and that points with high loss should correspond to a high gradient. Hence the dependency of the update of the parameters on the value of the loss should be already contained, although in a more indirect way. Is doing what I propose dangerous and/or useless?

  • $\begingroup$ Have you ever heard of adaptive learning rate? I think that's what you are looking for. $\endgroup$ – nbro Aug 6 '20 at 11:36
  • 1
    $\begingroup$ Yes, but I've never read of a learning rate scheduler that updates the learning rate explicitly depending on the value of the loss at each step. Usually, for example, you halven the learning rate at a plateau, or something similar. I want to change the learning rate at each step $\endgroup$ – Giulio Ortali Aug 6 '20 at 11:40
  • $\begingroup$ When you adjust your learning rate based on the loss, you have more chance to get stuck in local minima more often. It could be more efficient, but in general, I don't feel it has a positive effect in more complex situations. $\endgroup$ – Steven01123581321 Aug 6 '20 at 11:44
  • $\begingroup$ machinelearningmastery.com/… maybe this is a good resource to go to. $\endgroup$ – Steven01123581321 Aug 6 '20 at 11:49
  • $\begingroup$ Why you think you have more chances of getting a local minima? experience? $\endgroup$ – Giulio Ortali Aug 6 '20 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.