# Is it a good idea to change the learning rate at each training step as a function of the loss?

Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate (using a tailored function)?

I know that the update of the parameters is done via $$\gamma \nabla L$$, where $$\nabla L$$ is the gradient and $$\gamma$$ the learning rate, and that points with high loss should correspond to a high gradient. Hence the dependency of the update of the parameters on the value of the loss should be already contained, although in a more indirect way. Is doing what I propose dangerous and/or useless?

• Have you ever heard of adaptive learning rate? I think that's what you are looking for. – nbro Aug 6 '20 at 11:36
• Yes, but I've never read of a learning rate scheduler that updates the learning rate explicitly depending on the value of the loss at each step. Usually, for example, you halven the learning rate at a plateau, or something similar. I want to change the learning rate at each step – Giulio Ortali Aug 6 '20 at 11:40
• When you adjust your learning rate based on the loss, you have more chance to get stuck in local minima more often. It could be more efficient, but in general, I don't feel it has a positive effect in more complex situations. – Steven01123581321 Aug 6 '20 at 11:44
• machinelearningmastery.com/… maybe this is a good resource to go to. – Steven01123581321 Aug 6 '20 at 11:49
• Why you think you have more chances of getting a local minima? experience? – Giulio Ortali Aug 6 '20 at 12:02