Questions tagged [stochastic-gradient-descent]
For questions related to stochastic gradient descent (SGD), which is stochastic gradient descent that uses stochastic (or noisy) gradients.
8
questions with no upvoted or accepted answers
4
votes
0
answers
337
views
How does SGD escape local minima?
SGD is able to jump out of local minima that would otherwise trap BGD
I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
- 1,041
2
votes
0
answers
27
views
Methodologies for passing the best samples for a neural network to learn
Just an idea I am sure I read in a book some time ago, but I can't remember the name.
Given a very large dataset and a neural network (or anything that can learn via something like stochastic gradient ...
- 121
1
vote
0
answers
72
views
Why do momentum techniques not work well for RNNs?
AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
I've recently learned that it is not as helpful for RNNs, where ...
- 870
1
vote
0
answers
84
views
Why is the sample size of stochastic gradient descent a power of 2?
I watched the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses.
They take the sample size of the window to be $2^5 = 32$ or $2^6 ...
- 1,403
1
vote
0
answers
261
views
How does batch size affect model size?
I'm suffering from a significant brain fart while trying to get my head around how does batch size affect overall model size e.g for CNNs. Does it serve as an additional dimension for all the weight ...
- 161
0
votes
0
answers
13
views
Why to use gradient accumulation?
I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
- 101
0
votes
0
answers
18
views
Can we use Stochastic Gradient Descent and its derivatives for geographical researches?
I am a freshman AI engineering student actually working on a project where I have to analyze a big amount of geographical data to find out an optimal position of latitude and altitude for a specific ...
- 1
0
votes
0
answers
42
views
Why don't we use this intialization with SGD rather than random?
Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
- 126