I'm aware there are some optimizer such as Adam that adjust the learning rate for each dimension during training. However, afaik, the maximum learning rate they can have is still determined by the user's input.

So, I wonder if there are optimizers that can increase/decrease their overall learning rate and other parameters (such as momentum or even weight decay) autonomously depending on some metric, e.g., validation loss, running average of gradients etc. ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.