Questions tagged [stochastic-gradient-descent]
For questions related to stochastic gradient descent (SGD), which is stochastic gradient descent that uses stochastic (or noisy) gradients.
7
questions with no upvoted or accepted answers
4
votes
0answers
140 views
How does SGD escape local minima?
SGD is able to jump out of local minima that would otherwise trap BGD
I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
2
votes
0answers
27 views
Methodologies for passing the best samples for a neural network to learn
Just an idea I am sure I read in a book some time ago, but I can't remember the name.
Given a very large dataset and a neural network (or anything that can learn via something like stochastic gradient ...
2
votes
1answer
69 views
Why is the sample size of stochastic gradient descent a power of 2?
I watched in the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses.
They take the sample size of the window to be $2^5 = 32$ or $...
1
vote
0answers
64 views
Why do momentum techniques not work well for RNNs?
AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
I've recently learned that it is not as helpful for RNNs, where ...
0
votes
0answers
48 views
Commonly used learning rate schedules - linear warmup with linear decay?
I came across this post https://paperswithcode.com/methods/category/learning-rate-schedules which lists some different learning rate schedules and the number of papers which use them. I was a bit ...
0
votes
0answers
38 views
Why don't we use this intialization with SGD rather than random?
Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
0
votes
0answers
23 views
How is the marginal likelihood of wide valleys higher than that of narrow valleys when optimizing a cost function?
I am reading the paper Entropy-sgd: Biasing gradient descent into wide valleys by Chaudhari et al.
From what I understand, wide valleys tend to generalize better than sharp ones because they are ...
