13 votes
Accepted

Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
nbro's user avatar
  • 39.6k
12 votes
Accepted

What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
nagaK's user avatar
  • 271
11 votes
Accepted

Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
Neil Slater's user avatar
  • 30.4k
11 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
nbro's user avatar
  • 39.6k
11 votes
Accepted

Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
nbro's user avatar
  • 39.6k
10 votes

Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
Neil Slater's user avatar
  • 30.4k
9 votes
Accepted

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
nbro's user avatar
  • 39.6k
9 votes
Accepted

What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
Neil Slater's user avatar
  • 30.4k
8 votes
Accepted

Can non-differentiable layer be used in a neural network, if it's not learned?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ...
Neil Slater's user avatar
  • 30.4k
8 votes
Accepted

How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
NietzscheanAI's user avatar
7 votes
Accepted

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is \begin{align} \frac{d}{...
nbro's user avatar
  • 39.6k
7 votes

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$ They are not just "equivalent". It is actually the exact same function, with two different ways to write ...
Neil Slater's user avatar
  • 30.4k
7 votes
Accepted

Why is the perceptron criterion function differentiable?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
pasaba por aqui's user avatar
7 votes

Why is gradient descent used over the conjugate gradient method?

When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
Taw's user avatar
  • 1,211
6 votes
Accepted

Are filters fixed or learned?

What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ...
nbro's user avatar
  • 39.6k
6 votes

Is there an ideal range of learning rate which always gives a good result almost in all problems?

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the ...
Leland Hepworth's user avatar
5 votes
Accepted

Does Musk know what gradient descent is?

I think Musk was using the terminology correctly though perhaps with hyperbole. I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a ...
antlersoft's user avatar
5 votes

Is the mean-squared error always convex in the context of neural networks?

Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ...
varsh's user avatar
  • 562
5 votes
Accepted

Do good approximations produce good gradients?

In general $|f(x) - f_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either. You can also ...
Umang Gupta's user avatar
5 votes
Accepted

Deep Q-Learning: why don't we use mini-batches during experience reply?

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ...
Neil Slater's user avatar
  • 30.4k
5 votes
Accepted

Why evolutionary training of neural networks is not popular?

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers ...
John Doucette's user avatar
5 votes
Accepted

Eligibility vector for softmax policy with policy gradients

Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ...
Brale's user avatar
  • 2,336
5 votes
Accepted

Combine multiple losses with gradient descent

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--...
chessprogrammer's user avatar
4 votes

What does the notation $\nabla_\theta \mathcal{L}$ mean?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element ...
cantordust's user avatar
4 votes
Accepted

Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
Shrey's user avatar
  • 204
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
Nicola Bernini's user avatar
4 votes
Accepted

Can neuroevolution be combined with gradient descent?

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and ...
nbro's user avatar
  • 39.6k
4 votes
Accepted

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
Ketil Malde's user avatar
4 votes
Accepted

What is relation between gradient descent and regularization in deep learning?

Usually, when talking about regularization for neural networks there are 3 main types: L1, L2 and dropout. All affect the gradient descent procedure. L1 and L2 regularization is implemented in the ...
user10283726's user avatar
4 votes
Accepted

Why do we update all layers simultaneously while training a neural network?

Why do we update all layers simultaneously while training a neural network? We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm ...
nbro's user avatar
  • 39.6k

Only top scored, non community-wiki answers of a minimum length are eligible