It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my network keeps increasing.

From what I understand, what happens is that if I choose a learning rate that is too large, I overshoot the local minimum. But still, I am getting somewhere, and from there I'm moving in the correct direction. At worst my output should be random, I don't understand what is the scenario that causes my output and error to approach infinity every time I run my NN with a learning rate that is too large (and it's not even that large)

enter image description here How does the red line go to infinity ever? I kind of understand it could happen if we choose a crazy high learning rate, but if the NN works for 0.2 and doesn't for 0.4, I don't understand that

  • $\begingroup$ think of it in terms of weights. the weights would find a drastic change for example, from 1 to 1.001 for lr=0.001 or from 1 to 1.4 for lr=0.4. multiplied with any input, the latter would give a more drastic change/output (consider relu activated where inputs can be large). these accumulate over layers and increase the final output errors for next batch/sample after being updated by backprop and so, the backpropogated loss will also be high and the cycle continues until you get nan $\endgroup$
    – SajanGohil
    Jul 17 '20 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.