14 votes
Accepted

Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
nbro's user avatar
  • 40.6k
12 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
nbro's user avatar
  • 40.6k
11 votes
Accepted

Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
Neil Slater's user avatar
  • 32.1k
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
Dennis Soemers's user avatar
  • 10.3k
4 votes

Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i....
Yahya's user avatar
  • 141
4 votes

Why to use gradient accumulation?

This image from here nicely illustrates how gradient accumulation is performed: Assuming infinite memory and compute we would be able to compute the gradient on the full batch, this would provide us ...
Mariusmarten's user avatar
4 votes

What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

Is it possible to use Mini-Batches with Adam optimization?

Adam works best with mid-sized mini batches. Too small batches can generate too much sampling noise, making Adam less stable than basic stochastic gradient descent. Too large batches remove the ...
Neil Slater's user avatar
  • 32.1k
3 votes

What is the difference between batch and mini-batch gradient decent?

It is really simple. In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you ...
Gerry P's user avatar
  • 714
2 votes
Accepted

What's the rationale behind mini-batch gradient descent?

The basic idea behind mini-batch training is rooted in the exploration / exploitation tradeoff in local search and optimization algorithms. You can view training of an ANN as a local search through ...
John Doucette's user avatar
2 votes
Accepted

Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

There is a trade-off between the: Memory capacity of computation device Quality of gradient approximation Generalization ability of the network Memory capacity I would say, that it is possible to ...
spiridon_the_sun_rotator's user avatar
2 votes
Accepted

In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?

So, two things: Yes you technically calculate the accumulated error for each neuron, but in practice we calculate all neurons at once by treating it as a matrix instead of individual vectors. ...
Andy's user avatar
  • 471
1 vote
Accepted

How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?

You already mention quite some good ideas. Unfortunately, what makes GANs difficult to work with is their adversarial training paradigm. There are a ton of very simple and intricate ideas which might ...
Robin van Hoorn's user avatar
1 vote

What are your "current parameters" in Minibatch Stochastic Gradient Descent?

The current parameter values are the values of the weights $w$ and the biases $b$ in the neurons in each layer. What you are calculating is the gradient of the average loss of the batch relative to ...
David Hoelzer's user avatar
1 vote
Accepted

Does the summing or averaging of the weight gradients have anything to do with the cost function used?

What you stated looks correct :- The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed ...
Vivek Singhal's user avatar
1 vote
Accepted

Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the ...
spiridon_the_sun_rotator's user avatar
1 vote

When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?

Technically, nothing prevents you from doing so. When you have mulitple losses, you may call .backward() at each term separately. However, I wonder, whether it ...
spiridon_the_sun_rotator's user avatar
1 vote

When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?

I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each $L$ without running the algorithm for the mini-batch, it makes no ...
serali's user avatar
  • 890
1 vote
Accepted

In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have a special dimension (usually corresponding to the 0 axes) which numerates the example in the batch. Look for example ...
spiridon_the_sun_rotator's user avatar
1 vote
Accepted

When is the loss calculated, and when does the back-propagation take place?

Epoch: One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE. Batch Size: Total number of training examples present in a single batch. In real ...
mshlis's user avatar
  • 2,359
1 vote

Is neural networks training done one-by-one?

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge ...
Kartik Podugu's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible