Questions tagged [learning-rate]
For questions related to the concept of learning rate (of an optimization algorithm, such as gradient descent) in machine learning.
33
questions
0
votes
3
answers
59
views
My model is only improving when learning rate is 1. Should I be worried?
As the title says my GNN with three layers of GAT (Graph attention layers) is only moving the metrics when the learning rate is 1. As generally the learning rate is (0,1) should I be worried?
Also ...
- 1
1
vote
1
answer
214
views
Effects of hyperparameters in Q-learning
While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
Firstly, I noticed that increasing the ...
2
votes
0
answers
441
views
Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?
An article on https://spell.ml says
Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or ...
- 21
0
votes
1
answer
2k
views
Is learning rate the only reason for training loss oscillation after few epochs?
Consider the following loss curve
The x-axis is the no. of epochs and the y-axis is the loss function.
You can observe that loss is decreasing drastically for the first few epochs and then starts ...
- 3,612
0
votes
0
answers
280
views
GAN performance starts to get worse as training continues
I'm currently trying to train a GAN to recreate similar images from a dataset. The dataset is using the Eiffel Tower Pictures from Googles Quick Draw dataset. The images aren't very large (only 12x12 ...
2
votes
2
answers
710
views
How does $\alpha$ affect the convergence of the TD algorithm?
In Temporal-Difference Learning, we update our value function by $V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_{t}\right)\right)$
If we ...
- 143
2
votes
0
answers
78
views
Has the idea of using different learning rates for different layers been explored in the literature?
I wonder whether there are heuristic rules for the optimal selection of learning rates for different layers. I expect that there is no general recipe, but probably there are some choices that may be ...
- 2,564
2
votes
1
answer
281
views
How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:
Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$.
TD(0) method: ...
- 359
1
vote
1
answer
115
views
Can the optimal learning rate differ for different architectures?
In several courses and tutorials about neural networks, people often say that the learning rate (LR) should be the first hyper-parameter to be tuned before we tweak the others. For example, in this ...
- 1,240
1
vote
1
answer
98
views
Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?
I have the following results I am trying to make sense of. I have attached the loss curves here for reference.
As you can see, the first issue is that the validation loss is lower than the training ...
- 113
9
votes
2
answers
1k
views
Is there an ideal range of learning rate which always gives a good result almost in all problems?
I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph ...
- 208
1
vote
1
answer
166
views
Why is Adam trapped in bad/suspicious local optima after the first few updates?
In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write
To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
- 11
1
vote
1
answer
286
views
If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?
Q-Learning is guaranteed to converge if $\alpha$ decreases over time.
On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
- 737
4
votes
2
answers
100
views
Is stable learning preferable to jumps in accuracy/loss
A stable/smooth learning validation curve often seems to keep improving over more epochs than an unstable learning curve. My intuition is that dropping the learning rate and increasing the patience of ...
- 141
8
votes
1
answer
4k
views
Why is the learning rate generally beneath 1?
In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some ...
- 1,346
1
vote
0
answers
91
views
Is it a good idea to change the learning rate at each training step as a function of the loss?
Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate ...
3
votes
0
answers
81
views
Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?
I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
- 141
1
vote
1
answer
136
views
Why would the learning rate curve go backwards?
I'm working on recognizing the numbers 3 and 7 using the MNIST data set. I'm using cnn_learner() function from fastai library.
When I plotted the learning rate, the ...
- 121
1
vote
0
answers
198
views
How can a learning rate that is too large cause the output of the network (and the error) to go to infinity?
It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my ...
- 19
2
votes
0
answers
221
views
Why does learning rate reduce train-test generalization gap?
In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/
Prof Recht shows two plots:
He says one of the reasons the plot below has a lower train-test gap is because that model was trained ...
- 598
1
vote
0
answers
68
views
What is the equation of the learning rate decay in the Adam optimiser?
Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
- 1,251
1
vote
1
answer
349
views
Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?
It is known that if $\alpha$ is set to high, then the cost function of the model may not converge.
However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during ...
- 1,251
1
vote
1
answer
248
views
How to best make use of learning rate scheduling in reinforcement learning?
How to best make use of learning rate scheduling in reinforcement learning?
To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
- 265
0
votes
1
answer
185
views
Is it a good idea to overfit on a small part of your data for faster model convergence?
I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...
- 101
1
vote
3
answers
429
views
How to prove that gradient descent doesn't necessarily find the global optimum?
How can I prove that gradient descent doesn't necessarily find the global optimum?
For example, consider the following function
$$f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + ...
- 177
2
votes
1
answer
320
views
Autoencoder network for feature selection not converging
I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
- 51
1
vote
1
answer
352
views
Is this learning rate schedule increasing the learning rate?
I was reading a PyTorch code then I saw this learning rate scheduler:
...
- 185
3
votes
1
answer
3k
views
In Q-learning, shouldn't the learning rate change dynamically during the learning phase?
I have the following code (below), where an agent uses Q-learning (RL) to play a simple game.
What appears to be questionable for me in that code is the fixed learning rate. When it's set low, it's ...
- 279
9
votes
1
answer
1k
views
What causes a model to require a low learning rate?
I've pondered this for a while without developing an intuition for the math behind the cause of this.
So what causes a model to need a low learning rate?
- 217
2
votes
1
answer
2k
views
Why can the learning rate make the loss increase in stochastic gradient descent?
In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):
The main question is how to set $\epsilon_0$. If it is too large, the
...
- 204
1
vote
0
answers
118
views
What is the use of the $\epsilon$ term in this back-propagation equation?
I am currently looking at different documents to understand back-propagation, mainly at this document. Now, on page 3, there is the $\epsilon$ symbol involved:
$$
\Delta w_{k j}=\varepsilon \overbrace{...
- 425
6
votes
1
answer
2k
views
Should I be decaying the learning rate and the exploration rate in the same manner?
Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
- 91
4
votes
2
answers
3k
views
Is there a way to translate the concept of batch size into reinforcement learning?
I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up ...
- 41