Questions tagged [stochastic-gradient-descent]

For questions related to stochastic gradient descent (SGD), which is stochastic gradient descent that uses stochastic (or noisy) gradients.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

How does SGD escape local minima?

"SGD is able to jump out of local minima that would otherwise trap BGD" I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD(Stochastic ...
3
votes
2answers
101 views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
1
vote
1answer
49 views

Stochastic gradient descent does not behave as expected, even with different activation functions

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to ...
1
vote
1answer
22 views

What is the difference between batch and mini-batch gradient decent?

I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent. Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
1
vote
1answer
29 views

Why is the sample size of stochastic gradient descent a power of 2?

I watched in the video lecture of cs224: Stanford CS224N: NLP with Deep Learning | Winter 2019 | Lecture 2 – Word Vectors and Word Senses. They take the sample size of the window to be $2^5 = 32$ or $...
1
vote
1answer
39 views

How does batch size affect model size?

I'm suffering from a significant brain fart while trying to get my head around how does batch size affect overall model size e.g for CNNs. Does it serve as an additional dimension for all the weight ...