Questions tagged [mini-batch-gradient-descent]
For questions about mini-batch (or batch) gradient descent, which is gradient descent with typically more than one sample of input-label pairs.
3 questions with no upvoted or accepted answers
1
vote
0
answers
529
views
Why would one prefer the gradient of the sum rather than the sum of the gradients?
When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al.
$$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
- 11
0
votes
0
answers
28
views
Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy
I'm implementing batchnorm from scratch in pure NumPy.
I noticed something interesting.
While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
- 125
0
votes
0
answers
749
views
In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
