Skip to main content
New
Stack Overflow Jobs powered by Indeed: A job site that puts thousands of tech jobs at your fingertips (U.S. only). Search jobs

Questions tagged [mini-batch-gradient-descent]

For questions about mini-batch (or batch) gradient descent, which is gradient descent with typically more than one sample of input-label pairs.

Filter by
Sorted by
Tagged with
2 votes
1 answer
69 views

Is it possible to use Mini-Batches with Adam optimization?

Is it possible/advised to use Mini-Batch like accumulation with Adam optimization? How would that works? Do I accumulate the loss function for each sample in the batch and then run Adam, or should I ...
CoffeDeveloper's user avatar
0 votes
1 answer
119 views

In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?

In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the ...
will The J's user avatar
0 votes
1 answer
494 views

How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?

I am training several different GAN architectures, and I noticed that larger batch sizes may lead to vanishing or exploding gradients. In the interest of accelerating training, however, larger batch ...
postnubilaphoebus's user avatar
1 vote
1 answer
2k views

Why to use gradient accumulation?

I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
Cyrus's user avatar
  • 111
0 votes
1 answer
27 views

What are your "current parameters" in Minibatch Stochastic Gradient Descent?

I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense. Thus, we will often settle for sampling a random minibatch of examples every ...
HarshDarji's user avatar
1 vote
0 answers
454 views

Why would one prefer the gradient of the sum rather than the sum of the gradients?

When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al. $$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
Eddie C's user avatar
  • 11
0 votes
1 answer
200 views

Does the summing or averaging of the weight gradients have anything to do with the cost function used?

I've been trying to implement my own neural network library and have been wondering if: The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (...
jake_prentice's user avatar
1 vote
1 answer
119 views

Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
Robo's user avatar
  • 121
0 votes
0 answers
580 views

In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
Kaustubh Sharma's user avatar
1 vote
2 answers
80 views

When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?

I am training a neural network using a mini-batch gradient descent algorithm. Now, consider the following loss function, which is composed of 2 terms. $$L = L_{\text{MSE}} + L_{\text{regularization}} \...
hanugm's user avatar
  • 3,880
1 vote
1 answer
352 views

How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
user50018's user avatar
1 vote
2 answers
588 views

In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the stochastic gradient descent algorithm, the weight update happens for every training sample. In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
hanugm's user avatar
  • 3,880
2 votes
1 answer
489 views

Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent. For this question, assume that your dataset has $n$ ...
hanugm's user avatar
  • 3,880
1 vote
1 answer
137 views

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation. In my experiments, I am trying different ...
Bert Gayus's user avatar
10 votes
2 answers
5k views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
Ben's user avatar
  • 435
3 votes
2 answers
243 views

What is the difference between batch and mini-batch gradient decent?

I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent. Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
DRV's user avatar
  • 1,683
2 votes
1 answer
1k views

When is the loss calculated, and when does the back-propagation take place?

I read different articles and keep getting confused on this point. Not sure if the literature is giving mixed information or I'm interpreting it incorrectly. So from reading articles my understanding (...
Hazzaldo's user avatar
  • 279
10 votes
2 answers
1k views

Is neural networks training done one-by-one? [duplicate]

I'm trying to learn neural networks by watching this series of videos and implementing a simple neural network in Python. Here's one of the things I'm wondering about: I'm training the neural network ...
Ram Rachum's user avatar
10 votes
1 answer
11k views

Is back-propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
Maanu's user avatar
  • 245
2 votes
2 answers
182 views

What's the rationale behind mini-batch gradient descent?

I am reading a book that states As the mini-batch size increases, the gradient computed is closer to the 'true' gradient So, I assume that they are saying that mini-batch training only focuses on ...
ngc1300's user avatar
  • 133
3 votes
1 answer
445 views

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay. Do we update the parameters $\theta$ of function $Q$ once for all the samples of the ...
user491626's user avatar
1 vote
1 answer
917 views

What is the order of execution of steps in back-propagation algorithm in a neural network?

I am a machine learning newbie. I am trying to understand the back-propagation algorithm. I have a training dataset of 60 instances/records. What is the correct order of the process? This one? ...
gokul's user avatar
  • 53