The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it normal?

  • $\begingroup$ Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this. $\endgroup$
    – nbro
    Feb 29, 2020 at 12:53
  • $\begingroup$ @nbro, Yes, that true. The value distribution is within [-0.0001, 0.0001] $\endgroup$
    – GoingMyWay
    Feb 29, 2020 at 14:23
  • $\begingroup$ These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.? $\endgroup$
    – nbro
    Feb 29, 2020 at 16:24
  • 1
    $\begingroup$ Seems kinda arbitrary, really. If anything, you might think about scaling up your loss function to make it more sensitive. The main concern would be reducing harmful rounding due to the limited precision of floating-points. $\endgroup$
    – Nat
    Mar 2, 2020 at 13:44
  • $\begingroup$ @nbro at the early stage of training. I am using Normal initializer with mean=0 and std=1. $\endgroup$
    – GoingMyWay
    Mar 10, 2020 at 8:58


You must log in to answer this question.

Browse other questions tagged .