# Questions tagged [learning-rate]

For questions related to the concept of learning rate (of an optimization algorithm, such as gradient descent) in machine learning.

25 questions
Filter by
Sorted by
Tagged with
336 views

### Is there an ideal range of learning rate which always gives a good result almost in all problems?

I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph ...
63 views

### Is stable learning preferable to jumps in accuracy/loss

A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of ...
116 views

### Why is the learning rate generally beneath 1?

In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some ...
3k views

### Is there a way to translate the concept of batch size into reinforcement learning?

I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up ...
44 views

### Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
26 views

### How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function: Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$. TD(0) method: ...
133 views

### Autoencoder network for feature selection not converging

I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
52 views

### Why does learning rate reduce train-test generalization gap?

In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/ Prof Recht shows two plots: He says one of the reasons the plot below has a lower train-test gap is because that model was trained ...
69 views

### If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?

Q-Learning is guaranteed to converge if $\alpha$ decreases over time. On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
35 views

### Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

I have the following results I am trying to make sense of. I have attached the loss curves here for reference. As you can see, the first issue is that the validation loss is lower than the training ...
163 views

76 views

### How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
7 views

### Loss & accuracy curves from learning rate range test interpretation

I am working on a project doing experiments with the Learning Rate Range Test (See "A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and ...
38 views

### Commonly used learning rate schedules - linear warmup with linear decay?

I came across this post https://paperswithcode.com/methods/category/learning-rate-schedules which lists some different learning rate schedules and the number of papers which use them. I was a bit ...
22 views

### Should an increased learning rate for an adaptive linear neuron (ADALINE) reduce the square error at every epoch?

I am completely new to neural networks and therefore, my query may have some basic conceptual problem. I am following Fundamentals of Neural Networks by Laurene Fusett. In this book, the author ...