The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it normal?

  • $\begingroup$ Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this. $\endgroup$ – nbro Feb 29 '20 at 12:53
  • $\begingroup$ @nbro, Yes, that true. The value distribution is within [-0.0001, 0.0001] $\endgroup$ – GoingMyWay Feb 29 '20 at 14:23
  • $\begingroup$ These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.? $\endgroup$ – nbro Feb 29 '20 at 16:24
  • 1
    $\begingroup$ Seems kinda arbitrary, really. If anything, you might think about scaling up your loss function to make it more sensitive. The main concern would be reducing harmful rounding due to the limited precision of floating-points. $\endgroup$ – Nat Mar 2 '20 at 13:44
  • $\begingroup$ @nbro at the early stage of training. I am using Normal initializer with mean=0 and std=1. $\endgroup$ – GoingMyWay Mar 10 '20 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.