All Questions
9 questions
1
vote
1
answer
2k
views
Why to use gradient accumulation?
I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
- 111
1
vote
1
answer
148
views
Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?
Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
- 121
2
votes
1
answer
591
views
Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.
For this question, assume that your dataset has $n$ ...
- 3,990
11
votes
2
answers
6k
views
What exactly is averaged when doing batch gradient descent?
I have a question about how the averaging works when doing mini-batch gradient descent.
I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
- 445
3
votes
3
answers
278
views
What is the difference between batch and mini-batch gradient decent?
I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.
Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
- 1,763
10
votes
2
answers
1k
views
Is neural networks training done one-by-one? [duplicate]
I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python.
Here's one of the things I'm wondering about: I'm training the neural network ...
- 260
10
votes
1
answer
12k
views
Is back-propagation applied for each data point or for a batch of data points?
I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
- 245
2
votes
2
answers
187
views
What's the rationale behind mini-batch gradient descent?
I am reading a book that states
As the mini-batch size increases, the gradient computed is closer to the 'true' gradient
So, I assume that they are saying that mini-batch training only focuses on ...
- 133
1
vote
1
answer
1k
views
What is the order of execution of steps in back-propagation algorithm in a neural network?
I am a machine learning newbie. I am trying to understand the back-propagation algorithm. I have a training dataset of 60 instances/records.
What is the correct order of the process? This one?
...
- 53