13 votes
Accepted

Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
nbro's user avatar
  • 40.1k
12 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
nbro's user avatar
  • 40.1k
12 votes
Accepted

What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
nagaK's user avatar
  • 271
11 votes
Accepted

Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
Neil Slater's user avatar
  • 31.5k
11 votes
Accepted

Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
nbro's user avatar
  • 40.1k
9 votes
Accepted

Should we also shuffle the test dataset when training with SGD?

Short answer Shuffling affects learning (i.e. the updates of the parameters of the model), but, during testing or validation, you are not learning. So, it should not make any difference whether you ...
nbro's user avatar
  • 40.1k
7 votes

How do I choose the optimal batch size?

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch: If you have a small training set, use batch gradient descent (m < 200) In ...
Ayoub EL MAJJODI's user avatar
7 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
Tahlor's user avatar
  • 171
6 votes

How do I choose the optimal batch size?

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee. How to Configure Mini-Batch Gradient Descent Mini-batch gradient descent ...
4 votes

How do I choose the optimal batch size?

The batch size can also have a significant impact on your model’s performance and the training time. In general, the optimal batch size will be lower than 32. In April 2018, Yann Lecun even tweeted: ...
Om. Sa223's user avatar
4 votes

What exactly is averaged when doing batch gradient descent?

do I have to: forward propagate calculate error calculate all gradients ...repeatedly over all samples in the batch, and then average all gradients and apply the weight change? Yes, that is ...
Neil Slater's user avatar
  • 31.5k
4 votes

Why to use gradient accumulation?

This image from here nicely illustrates how gradient accumulation is performed: Assuming infinite memory and compute we would be able to compute the gradient on the full batch, this would provide us ...
Mariusmarten's user avatar
4 votes

Is there any way to train a neural network without using gradients?

Yes. A prominent class of "gradient-free" algorithms in ML world is known as Evolution Strategies (ES). Evolutionary Algorithms, although existed for a long time, only a few have shown to ...
ayandas's user avatar
  • 238
4 votes

Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

You are correct, but requires final words: In Batch GD, we take the average of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i....
Yahya's user avatar
  • 141
4 votes
Accepted

How are these equations of SGD with momentum equivalent?

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately. Equation 1 Consider the equation from the Stanford slide: $$ v_{t}=\rho v_{t-1}+\nabla ...
user3667125's user avatar
  • 1,530
3 votes

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many ...
David's user avatar
  • 4,760
3 votes

What is the difference between batch and mini-batch gradient decent?

It is really simple. In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you ...
Gerry P's user avatar
  • 714
3 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
user3711746's user avatar
2 votes
Accepted

What's the rationale behind mini-batch gradient descent?

The basic idea behind mini-batch training is rooted in the exploration / exploitation tradeoff in local search and optimization algorithms. You can view training of an ANN as a local search through ...
John Doucette's user avatar
2 votes
Accepted

In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?

The lower bound in MINE is as follows: $$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}...
Mohith Kaameswaran Sakthivel's user avatar
2 votes
Accepted

Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

There is a trade-off between the: Memory capacity of computation device Quality of gradient approximation Generalization ability of the network Memory capacity I would say, that it is possible to ...
spiridon_the_sun_rotator's user avatar
1 vote

What does it mean if I trained my model with more steps per epoch than the total number of training images I have?

Welcome to AI.stackexchange! To answer your question more precisely, it would be helpful to provide a minimum working example of your code where we can see how you implemented your training loop. ...
tanasr's user avatar
  • 72
1 vote
Accepted

Can MSE be used for NN categorical classification problems

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems. A theoretical reason Both aforementioned losses are negative logarithmic ...
Vladimir Yankovskiy's user avatar
1 vote

How does SGD training error decrease in subsequent epochs when statistically, it requires that samples in subsequent epochs be i.i.d and they are not?

The samples might be the same in each epoch, but, before starting the new epoch, the training dataset might be randomly shuffled before splitting it into mini-batches (this is the default behaviour in ...
nbro's user avatar
  • 40.1k
1 vote
Accepted

Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia

That paragraph is incomplete and unclear indeed. Let's crack the difference with a concrete example: logistic regression. The objective we want to minimize is: $J_{train}(\theta)=\frac{1}{2m}\sum_{i=1}...
Edoardo Guerriero's user avatar
1 vote

Issue with graphical interpretation of the universal approximation theorem

The classical version of the universal approximation theorem states that, roughly, given a continuous function $f \colon [0, 1]^n \to [0, 1]^n$, there exists a single layer neural network and a set of ...
htl's user avatar
  • 1,000
1 vote

Stochastic gradient descent does not behave as expected, even with different activation functions

Could you post the pseudocode of your backpropagation algorithm? I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it ...
david david's user avatar
1 vote

Is the choice of the optimiser relevant when doing object detection?

I have experimented with this to a small degree and have not noticed that much of an impact. To date, Adam appears to give the best results on a variety of image data sets. I have found that "...
Gerry P's user avatar
  • 714
1 vote

Is neural networks training done one-by-one?

Ideally, you need to update weights by going over all the samples in the dataset. This is called as Batch Gradient Descent. But, as the no. of training examples increases, the computation becomes huge ...
Kartik Podugu's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible