5

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, does not hold (given that a number bigger than $1$ squared becomes a bigger number), so stochastic gradient descent is not generally guaranteed to converge to a ...


5

Why is this a convergence criterion? It is because $R$ and $S'$ are stochastic. A large learning rate applied when these values have variance would not converge to mean, but would wander around typically within some value proportional to $\alpha\sigma$ of the true value, where $\sigma$ is the standard deviation of the term $R + \gamma\text{max}_aQ(S',a)$. ...


5

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed ...


4

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms. In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient ...


3

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations: Data from the training set could be identical or leaking in to the validation set. Using a high dropout rate can cause this as well as ...


3

It should be noted that the selection of $\alpha$ is a classic problem in stochastic approximation, rather than a specific problem in RL. Once you know this it will be clear. What is stochastic approximation? As its name suggests, it is a method that uses data to approximate (typically) expectations. For example, suppose \begin{align*} w=\mathbb{E}[X] \end{...


3

Potentially. If you do offline reinforcement learning, you're basically learning to approximate a function by sampling input/output pairs, rather than episode-by-episode. Here, your batch size could be set exactly as in an ordinary supervised learning problem. If you do online learning, then it's not clear to me that the techniques used to set the learning ...


3

Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments. However, this is something that is harder to manage than in supervised learning, and might not be as useful or necessary. The issues you need to be concerned about are: Non-stationarity The target values in value-...


2

You can find by yourself a counterexample that, in general, GD is not guaranteed to find the global optimum! I first advise you to choose a simpler function (than the one you are showing), with 2-3 optima, where one is the global and the other(s) are local. You don't need neural networks or any other ML concept to show this, but only basic calculus (...


2

There is an approach to machine learning, called Simulated Annealing, which varies the rate: starting from a large rate, it is slowly reduced over time. The general idea is that the initial larger rate will cover a broader range, while the increasingly lower rate then produces a less 'erratic' climb towards a maximum. If you only use a low rate, you risk ...


2

So why is constant-$\alpha$ being used? This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly. From my understanding, in stationary environments, ...


1

Yes, the optimal learning rate will differ for every change you make in the network. In fact finding the optimal learning rate is very computationally expensive, so you will normally only get a rough guess anyway. The learning rate is used to traverse an N dimensional loss landscape that changes drastically with even the smallest differences. If you add one ...


1

If you have an erratic loss landscape, it can lead to an unstable learning curve. Thus, it's always better to choose a simpler function which creates a simple landscape. Sometimes even due to uneven dataset distribution, we can observe those jumps/irregularities in the training curve. And yes, those jumps do mean it might've found something significant in ...


1

I have not used fastai library but this also happens on tensorboard when you have more than one training being recorded on the same plot. Looking at the picture, I think this is a very special type of graph because for a single LR value you have 2 loss values associated. Put in other words, you have the same LR value for different loss values. My guess it ...


1

Setting too high a learning rate will extend the time to get a good result. In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than ...


1

Well, GD terminates once the gradients are 0, right? Now, in a non-convex function, there could be some points, which do not belong to the global minima, and yet, have 0 gradients. For example, such points can belong to saddle points and local minima. Consider this picture and say you start GD at the x label. GD will bring you the flat area and will stop ...


1

The trick was to normalize the input dataset values with the respective mean and standard deviation in each column. This reduced the loss drastically, and my network is training more efficiently now. Moreover, normalizing the data also helps you calculate the weights associated with each input node more easily, especially when trying to find out variable ...


1

The higher (or smaller) the learning rate, the higher (or, respectively, smaller) the contribution of the gradient of the objective function, with respect to the parameters of the model, to the new parameters of the model. Therefore, if you progressively increase (or decrease) the learning rate, then you will accelerate (or, respectively, slow down) the ...


1

From my understanding of reinforcement learning, you will have an agent and an environment. In each episode, the agent observes the state $s$, takes some action action $a$, then gets some reward $r$, and finally observes the next state $s'$, and do it again and again until the end of the episode. The above process does not incur any "learning". ...


Only top voted, non community-wiki answers of a minimum length are eligible