Questions tagged [learning-rate]

For questions related to the concept of learning rate (of an optimization algorithm, such as gradient descent) in machine learning.

10 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
43 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
2
votes
0answers
48 views

Why does learning rate reduce train-test generalization gap?

In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/ Prof Recht shows two plots: He says one of the reasons the plot below has a lower train-test gap is because that model was trained ...
1
vote
1answer
34 views

Why is Adam trapped in bad/suspicious local optima after the first few updates?

In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write To further analyze this phenomenon, we visualize the histogram of the absolute value of ...
1
vote
0answers
43 views

Is it a good idea to change the learning rate at each training step as a function of the loss?

Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate ...
1
vote
0answers
39 views

How can a learning rate that is too large cause the output of the network (and the error) to go to infinity?

It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my ...
1
vote
0answers
30 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
1
vote
1answer
72 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
0
votes
0answers
22 views

Should an increased learning rate for an adaptive linear neuron (ADALINE) reduce the square error at every epoch?

I am completely new to neural networks and therefore, my query may have some basic conceptual problem. I am following Fundamentals of Neural Networks by Laurene Fusett. In this book, the author ...
0
votes
0answers
126 views

How the parameters of decay_rate & decay_steps are taken into consideration while computing InverseTimeDecay()

During the last couple of days, I am experimenting with the different schedulers of learning rate decay offered by Keras (link here). Specifically, I have been using InverseTimeDecay: A ...
0
votes
1answer
63 views

Is it a good idea to overfit on a small part of your data for faster model convergence?

I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...