# Tag Info

Accepted

### Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
• 26.5k

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
• 37k
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
• 37k
Accepted

### What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
• 241
Accepted

### Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
• 37k

### How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
• 171

### How do I choose the optimal batch size?

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent ...

### How do I choose the optimal batch size?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In ...
Accepted

### Should we also shuffle the test dataset when training with SGD?

Short answer Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you ...
• 37k

### Is there any way to train a neural network without using gradients?

Yes. A prominent class of "gradient-free" algorithms in ML world is known as Evolution Strategies (ES). Evolutionary Algorithms, although existed for a long time, only a few have shown to ...
• 238

### Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i....
• 141

### What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
• 26.5k
Accepted

Accepted

### Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

There is a trade-off between the: Memory capacity of computation device Quality of gradient approximation Generalization ability of the network Memory capacity I would say, that it is possible to ...
1 vote
Accepted

### Can MSE be used for NN categorical classification problems

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems. A theoretical reason Both aforementioned losses are negative logarithmic ...
1 vote

### How does SGD training error decrease in subsequent epochs when statistically, it requires that samples in subsequent epochs be i.i.d and they are not?

The samples might be the same in each epoch, but, before starting the new epoch, the training dataset might be randomly shuffled before splitting it into mini-batches (this is the default behaviour in ...
• 37k
1 vote
Accepted

• 1,000
1 vote

### Stochastic gradient descent does not behave as expected, even with different activation functions

Could you post the pseudocode of your backpropagation algorithm? I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it ...
• 102
1 vote

### Is the choice of the optimiser relevant when doing object detection?

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
• 694
1 vote

### Is neural networks training done one-by-one?

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge ...

Only top scored, non community-wiki answers of a minimum length are eligible