Questions tagged [mini-batch-gradient-descent]
For questions about mini-batch (or batch) gradient descent, which is gradient descent with typically more than one sample of input-label pairs.
24 questions
1
vote
1
answer
26
views
How to normalize gradient value due to the batch size?
A = (m x n) - input
B = (n x k) - weight
output = A @ B = (m x k)
...
- 39
0
votes
0
answers
28
views
Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy
I'm implementing batchnorm from scratch in pure NumPy.
I noticed something interesting.
While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
- 125
2
votes
1
answer
322
views
Is it possible to use Mini-Batches with Adam optimization?
Is it possible/advised to use Mini-Batch like accumulation with Adam optimization?
How would that works?
Do I accumulate the loss function for each sample in the batch and then run Adam, or should I ...
- 301
0
votes
1
answer
164
views
In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?
In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the ...
- 267
0
votes
1
answer
782
views
How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?
I am training several different GAN architectures, and I noticed that larger batch sizes may lead to vanishing or exploding gradients. In the interest of accelerating training, however, larger batch ...
1
vote
1
answer
2k
views
Why to use gradient accumulation?
I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
- 111
0
votes
1
answer
32
views
What are your "current parameters" in Minibatch Stochastic Gradient Descent?
I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense.
Thus, we will often settle for sampling a random minibatch of examples every ...
- 101
1
vote
0
answers
529
views
Why would one prefer the gradient of the sum rather than the sum of the gradients?
When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al.
$$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
- 11
0
votes
1
answer
220
views
Does the summing or averaging of the weight gradients have anything to do with the cost function used?
I've been trying to implement my own neural network library and have been wondering if:
The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (...
1
vote
1
answer
148
views
Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?
Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
- 121
0
votes
0
answers
749
views
In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
1
vote
2
answers
82
views
When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?
I am training a neural network using a mini-batch gradient descent algorithm.
Now, consider the following loss function, which is composed of 2 terms.
$$L = L_{\text{MSE}} + L_{\text{regularization}} \...
- 3,990
1
vote
1
answer
407
views
How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
- 13
1
vote
2
answers
721
views
In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?
In the stochastic gradient descent algorithm, the weight update happens for every training sample.
In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
- 3,990
2
votes
1
answer
591
views
Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.
For this question, assume that your dataset has $n$ ...
- 3,990
1
vote
1
answer
144
views
Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation.
In my experiments, I am trying different ...
- 635
11
votes
2
answers
6k
views
What exactly is averaged when doing batch gradient descent?
I have a question about how the averaging works when doing mini-batch gradient descent.
I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
- 445
3
votes
3
answers
278
views
What is the difference between batch and mini-batch gradient decent?
I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.
Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
- 1,763
2
votes
1
answer
2k
views
When is the loss calculated, and when does the back-propagation take place?
I read different articles and keep getting confused on this point. Not sure if the literature is giving mixed information or I'm interpreting it incorrectly.
So from reading articles my understanding (...
- 299
10
votes
2
answers
1k
views
Is neural networks training done one-by-one? [duplicate]
I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python.
Here's one of the things I'm wondering about: I'm training the neural network ...
- 260
10
votes
1
answer
12k
views
Is back-propagation applied for each data point or for a batch of data points?
I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
- 245
2
votes
2
answers
187
views
What's the rationale behind mini-batch gradient descent?
I am reading a book that states
As the mini-batch size increases, the gradient computed is closer to the 'true' gradient
So, I assume that they are saying that mini-batch training only focuses on ...
- 133
3
votes
1
answer
452
views
When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay.
Do we update the parameters $\theta$ of function $Q$ once for all the samples of the ...
- 241
1
vote
1
answer
1k
views
What is the order of execution of steps in back-propagation algorithm in a neural network?
I am a machine learning newbie. I am trying to understand the back-propagation algorithm. I have a training dataset of 60 instances/records.
What is the correct order of the process? This one?
...
- 53