13
votes
Accepted
Is back-propagation applied for each data point or for a batch of data points?
Short answers
Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?
You can perform back-propagation using (or after) ...
- 39.6k
12
votes
Accepted
What is the relationship between gradient accumulation and batch size?
There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
- 271
11
votes
Accepted
Is neural networks training done one-by-one?
Should I be changing the weights/biases on every single sample before moving on to the next sample,
You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
- 30.4k
11
votes
What exactly is averaged when doing batch gradient descent?
Introduction
First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
- 39.6k
11
votes
Accepted
Why is the learning rate generally beneath 1?
If the learning rate is greater than or equal to $1$ the Robbins-Monro condition
$$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$
where $a_t$ is the learning rate at iteration $t$, ...
- 39.6k
10
votes
Can the mean squared error be negative?
In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.
A standard Mean Squared Error function cannot be ...
- 30.4k
9
votes
Accepted
How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?
$g(x) = x^2$ is indeed a parabola and thus has just one optimum.
However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
- 39.6k
9
votes
Accepted
What is the formula for the momentum and Adam optimisers?
I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
- 30.4k
8
votes
Accepted
Can non-differentiable layer be used in a neural network, if it's not learned?
It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so.
For example:
If an ...
- 30.4k
8
votes
Accepted
How to avoid falling into the "local minima" trap?
There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include:
Probabalistically accepting worse solutions in the hope that this
will ...
- 7,226
7
votes
Accepted
Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?
The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is
\begin{align}
\frac{d}{...
- 39.6k
7
votes
Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?
The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$
They are not just "equivalent". It is actually the exact same function, with two different ways to write ...
- 30.4k
7
votes
Accepted
Why is the perceptron criterion function differentiable?
$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$.
Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
- 1,283
7
votes
Why is gradient descent used over the conjugate gradient method?
When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
- 1,211
6
votes
Accepted
Are filters fixed or learned?
What are filters in image processing?
In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ...
- 39.6k
6
votes
Is there an ideal range of learning rate which always gives a good result almost in all problems?
The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
The paper's primary focus is the ...
- 161
5
votes
Accepted
Does Musk know what gradient descent is?
I think Musk was using the terminology correctly though perhaps with hyperbole.
I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a ...
- 617
5
votes
Is the mean-squared error always convex in the context of neural networks?
Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ...
- 562
5
votes
Accepted
Do good approximations produce good gradients?
In general $|f(x) - f_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either.
You can also ...
- 200
5
votes
Accepted
Deep Q-Learning: why don't we use mini-batches during experience reply?
DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ...
- 30.4k
5
votes
Accepted
Why evolutionary training of neural networks is not popular?
The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers ...
- 9,207
5
votes
Accepted
Eligibility vector for softmax policy with policy gradients
Calculation of gradient
\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
&= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}
is only ...
- 2,336
5
votes
Accepted
Combine multiple losses with gradient descent
This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--...
- 2,355
4
votes
What does the notation $\nabla_\theta \mathcal{L}$ mean?
This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element ...
- 943
4
votes
Accepted
Can gradient descent training be used for non-smooth loss functions?
Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
- 204
4
votes
Accepted
Could error surface shape be useful to detect which local minima is better for generalization?
In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations:
what the author of the question provides us with is the Loss ...
- 799
4
votes
Accepted
Can neuroevolution be combined with gradient descent?
The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and ...
- 39.6k
4
votes
Accepted
Is there a reason to choose regular momentum over Nesterov momentum for neural networks?
The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that
Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
- 224
4
votes
Accepted
What is relation between gradient descent and regularization in deep learning?
Usually, when talking about regularization for neural networks there are 3 main types:
L1, L2 and dropout. All affect the gradient descent procedure.
L1 and L2 regularization is implemented in the ...
- 68
4
votes
Accepted
Why do we update all layers simultaneously while training a neural network?
Why do we update all layers simultaneously while training a neural network?
We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm ...
- 39.6k
Only top scored, non community-wiki answers of a minimum length are eligible