Questions tagged [stochastic-gradient-descent]

For questions related to stochastic gradient descent (SGD), which is stochastic gradient descent that uses stochastic (or noisy) gradients.

7 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
140 views

How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
2
votes
0answers
27 views

Methodologies for passing the best samples for a neural network to learn

Just an idea I am sure I read in a book some time ago, but I can't remember the name. Given a very large dataset and a neural network (or anything that can learn via something like stochastic gradient ...
2
votes
1answer
69 views

Why is the sample size of stochastic gradient descent a power of 2?

I watched in the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses. They take the sample size of the window to be $2^5 = 32$ or $...
1
vote
0answers
64 views

Why do momentum techniques not work well for RNNs?

AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy. I've recently learned that it is not as helpful for RNNs, where ...
0
votes
0answers
48 views

Commonly used learning rate schedules - linear warmup with linear decay?

I came across this post https://paperswithcode.com/methods/category/learning-rate-schedules which lists some different learning rate schedules and the number of papers which use them. I was a bit ...
0
votes
0answers
38 views

Why don't we use this intialization with SGD rather than random?

Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
0
votes
0answers
23 views

How is the marginal likelihood of wide valleys higher than that of narrow valleys when optimizing a cost function?

I am reading the paper Entropy-sgd: Biasing gradient descent into wide valleys by Chaudhari et al. From what I understand, wide valleys tend to generalize better than sharp ones because they are ...