5

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed ...


4

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms. In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient ...


3

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations: Data from the training set could be identical or leaking in to the validation set. Using a high dropout rate can cause this as well as ...


2

So why is constant-$\alpha$ being used? This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly. From my understanding, in stationary environments, ...


1

Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway. The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one ...


1

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction. For example: Let $L(\theta) = \theta^2 $ and $\theta_0= 3$ Let the subscript n in $\theta_n$ denote the iteration number. Then $\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$ For the loss to decrease in this case $\epsilon < 1$ needs to hold ...


Only top voted, non community-wiki answers of a minimum length are eligible