# Questions tagged [learning-rate]

For questions related to the concept of learning rate (of an optimization algorithm, such as gradient descent) in machine learning.

26 questions
Filter by
Sorted by
Tagged with
9 views

### Loss & accuracy curves from learning rate range test interpretation

I am working on a project doing experiments with the Learning Rate Range Test (See "A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and ...
43 views

### Commonly used learning rate schedules - linear warmup with linear decay?

I came across this post https://paperswithcode.com/methods/category/learning-rate-schedules which lists some different learning rate schedules and the number of papers which use them. I was a bit ...
26 views

### How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function: Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$. TD(0) method: ...
41 views

### Can the optimal learning rate differ for different architectures?

In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
35 views

### Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

I have the following results I am trying to make sense of. I have attached the loss curves here for reference. As you can see, the first issue is that the validation loss is lower than the training ...
22 views

### Should an increased learning rate for an adaptive linear neuron (ADALINE) reduce the square error at every epoch?

I am completely new to neural networks and therefore, my query may have some basic conceptual problem. I am following Fundamentals of Neural Networks by Laurene Fusett. In this book, the author ...
336 views

### Is there an ideal range of learning rate which always gives a good result almost in all problems?

I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph ...
162 views

### How the parameters of decay_rate & decay_steps are taken into consideration while computing InverseTimeDecay()

During the last couple of days, I am experimenting with the different schedulers of learning rate decay offered by Keras (link here). Specifically, I have been using InverseTimeDecay: A ...
44 views

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
69 views

### If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?

Q-Learning is guaranteed to converge if $\alpha$ decreases over time. On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
63 views

### Is stable learning preferable to jumps in accuracy/loss

A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of ...
185 views

### Why is the learning rate generally beneath 1?

In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some ...
44 views

### Is it a good idea to change the learning rate at each training step as a function of the loss?

Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate ...
44 views

### Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
92 views

### Why would the learning rate curve go backwards?

I'm working on recognizing the numbers 3 and 7 using the MNIST data set. I'm using cnn_learner() function from fastai library. When I plotted the learning rate, the ...
40 views

### How can a learning rate that is too large cause the output of the network (and the error) to go to infinity?

It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my ...
53 views

### Why does learning rate reduce train-test generalization gap?

In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/ Prof Recht shows two plots: He says one of the reasons the plot below has a lower train-test gap is because that model was trained ...
30 views

### What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2$$ Hence, ...
38 views

### Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?

It is known that if $\alpha$ is set to high, then the cost function of the model may not converge. However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during ...
79 views

### How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
68 views

### Is it a good idea to overfit on a small part of your data for faster model convergence?

I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...
166 views

### How to prove that gradient descent doesn't necessarily find the global optimum?

How can I prove that gradient descent doesn't necessarily find the global optimum? For example, consider the following function f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + ...
137 views

### Autoencoder network for feature selection not converging

I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
140 views

### Is this learning rate schedule increasing the learning rate?

I was reading a PyTorch code then I saw this learning rate scheduler: ...