I'm learning Neural Networks, and everything works as planned but, like humans do, adjusting themselves to learn more efficiently, I'm trying to understand conceptually how one might implement an auto adjusting learning rate for a Neural Network.

I have tried to make it based on error, something like how bigger is error learning rate is getting bigger as well. [Could use some clarification here--not entirely sure what you're saying. If can clarify, I'm happy to clean up the English. -DukeZhou]

*If you want give me an example give it on a C based language or math because I don't have experience with Python or Pascal.

  • 1
    $\begingroup$ Google "adaptive learning rate" $\endgroup$
    – SmallChess
    Jul 31, 2017 at 1:10
  • $\begingroup$ cs231n.github.io/neural-networks-3/#ada $\endgroup$
    – SmallChess
    Jul 31, 2017 at 1:11
  • $\begingroup$ thx, you can make this comment an answer to mark it as best answer? $\endgroup$ Jul 31, 2017 at 10:49
  • $\begingroup$ @SmallChess very few people seem to be voting right now, but I'd definitely upvote a formal answer! $\endgroup$
    – DukeZhou
    Jul 31, 2017 at 18:50

3 Answers 3


Adjusting for learning rate is a common scenario in machine learning. There is rich literature about it, countless papers. The most common implementation:

  • AdaGrad
  • RMSProp
  • Adam

There are many more variants. You'll need to do some research. Please take a look at:


enter image description here


Apart from the already mentioned methods there is one which I came across recently is the Cyclic Learning Rates.

Mainstream methods usually reduce the learning rates monotonically. In most of the cases they work well. Yet they do not do well with handling saddle points and local minima. CLR method claims to solve the problem by effectively handling the case of saddle points.

Ref: https://arxiv.org/abs/1506.01186


While adjusting the learning rate during the training is certainly interesting, so is finding of a good initial learning rate, too.

Cyclical Learning Rates for Training Neural Networks, fast.ai (pytorch) implementation.

Here's a good practitioner's overview of learning rate schedules (python, but rather readable and with nice plots).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .