Questions tagged [learning-rate]

For questions related to the concept of learning rate (of an optimization algorithm, such as gradient descent) in machine learning.

Filter by
Sorted by
Tagged with
1
vote
1answer
33 views

Why is the learning rate generally beneath 1?

In all examples I've ever seen the learning rate of an optimisation method is always < 1. However, I've never found an explanation as to why this is. In addition to that, there are some cases where ...
0
votes
0answers
39 views

Is it a good idea to change the learning rate at each training step as a function of the loss?

Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate ...
2
votes
0answers
37 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
1
vote
1answer
63 views

Why would the learning rate curve go backwards?

I'm working on recognizing the numbers 3 and 7 using the MNIST data set. I'm using cnn_learner() function from fastai library. When I plotted the learning rate, the ...
1
vote
0answers
33 views

How can a learning rate that is too large cause the output of the network (and the error) to go to infinity?

It happened to my neural network, when I use a learning rate of <0.2 everything works fine, but when I try something above 0.4 I start getting "nan" errors because the output of my ...
2
votes
0answers
37 views

Why does learning rate reduce train-test generalization gap?

In this blog post: http://www.argmin.net/2016/04/18/bottoming-out/ Prof Recht shows two plots: He says one of the reasons the plot below has a lower train-test gap is because that model was trained ...
1
vote
0answers
27 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
1
vote
1answer
29 views

Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?

It is known that if $\alpha$ is set to high, then the cost function of the model may not converge. However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during ...
1
vote
1answer
49 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
0
votes
1answer
45 views

Is it a good idea to overfit on a small part of your data for faster model convergence?

I working on a classification problem that needs to detect patterns on a time serie. Basically, there's a catch-all class that means "no pattern detected", the other are for the specific patterns. The ...
0
votes
0answers
40 views

Are there any papers on this alternate neural net training approach?

I developed a custom callback for Keras. Initially, it monitors training accuracy. If on a given epoch the accuracy is below that of the previous epoch it lowers the learning rate by a factor. If for ...
1
vote
3answers
103 views

How to prove that gradient descent doesn't necessarily find the global optimum?

How can I prove that gradient descent doesn't necessarily find the global optimum? For example, consider the following function $$f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + ...
2
votes
1answer
47 views

Autoencoder network for feature selection not converging

I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For ...
1
vote
1answer
79 views

Is this learning rate schedule increasing the learning rate?

I was reading a PyTorch code then I saw this learning rate scheduler: ...
3
votes
2answers
2k views

Is there a way to translate the concept of batch size into reinforcement learning?

I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up ...