I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.

I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. [Could use some clarification here--not entirely sure what you're saying. If can clarify, I'm happy to clean up the English. -DukeZhou]

*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal.

  • 1
    $\begingroup$ Google "adaptive learning rate" $\endgroup$ – HelloWorld Jul 31 '17 at 1:10
  • $\begingroup$ cs231n.github.io/neural-networks-3/#ada $\endgroup$ – HelloWorld Jul 31 '17 at 1:11
  • $\begingroup$ thx, you can make this comment an answer to mark it as best answer? $\endgroup$ – Laceanu George Jul 31 '17 at 10:49
  • $\begingroup$ @SmallChess very few people seem to be voting right now, but I'd definitely upvote a formal answer! $\endgroup$ – DukeZhou Jul 31 '17 at 18:50

Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:

  • AdaGrad
  • RMSProp
  • Adam

There are many more variants. You'll need to do some research. Please take a look at:


enter image description here


Apart from the already mentioned methods there is one which I came across recently is the Cyclic Learning Rates.

Mainstream methods usually reduce the learning rates monotonically. In most of the cases they work well. Yet they do not do well with handling saddle points and local minima. CLR method claims to solve the problem by effectively handling the case of saddle points.

Ref: https://arxiv.org/abs/1506.01186


While adjusting the learning rate during the training is certainly interesting, so is finding of a good initial learning rate, too.

Cyclical Learning Rates for Training Neural Networks, fast.ai (pytorch) implementation.

Here's a good practitioner's overview of learning rate schedules (python, but rather readable and with nice plots).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.