# Tag Info

Accepted

### Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
• 40.1k

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
• 40.1k
Accepted

### What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
• 271
Accepted

### Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
• 31.5k
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
• 40.1k
Accepted

### Should we also shuffle the test dataset when training with SGD?

Short answer Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you ...
• 40.1k

### How do I choose the optimal batch size?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In ...

### How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
• 171

### How do I choose the optimal batch size?

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent ...

### How do I choose the optimal batch size?

The batch size can also have a significant impact on your model’s performance and the training time. In general, the optimal batch size will be lower than 32. In April 2018, Yann Lecun even tweeted: ...

### What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
• 31.5k

### Why to use gradient accumulation?

This image from here nicely illustrates how gradient accumulation is performed: Assuming infinite memory and compute we would be able to compute the gradient on the full batch, this would provide us ...
• 433

### Is there any way to train a neural network without using gradients?

Yes. A prominent class of "gradient-free" algorithms in ML world is known as Evolution Strategies (ES). Evolutionary Algorithms, although existed for a long time, only a few have shown to ...
• 238

### Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i....
• 141
Accepted

Accepted

### Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

There is a trade-off between the: Memory capacity of computation device Quality of gradient approximation Generalization ability of the network Memory capacity I would say, that it is possible to ...
1 vote

### What does it mean if I trained my model with more steps per epoch than the total number of training images I have?

Welcome to AI.stackexchange! To answer your question more precisely, it would be helpful to provide a minimum working example of your code where we can see how you implemented your training loop. ...
• 72
1 vote
Accepted

### Can MSE be used for NN categorical classification problems

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems. A theoretical reason Both aforementioned losses are negative logarithmic ...
1 vote

### How does SGD training error decrease in subsequent epochs when statistically, it requires that samples in subsequent epochs be i.i.d and they are not?

The samples might be the same in each epoch, but, before starting the new epoch, the training dataset might be randomly shuffled before splitting it into mini-batches (this is the default behaviour in ...
• 40.1k
1 vote
Accepted

• 1,000
1 vote

### Stochastic gradient descent does not behave as expected, even with different activation functions

Could you post the pseudocode of your backpropagation algorithm? I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it ...
• 212
1 vote

### Is the choice of the optimiser relevant when doing object detection?

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
• 714
1 vote

### Is neural networks training done one-by-one?

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge ...

Only top scored, non community-wiki answers of a minimum length are eligible