11 votes
Accepted

Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
user avatar
  • 23.8k
9 votes
Accepted

What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
user avatar
  • 23.8k
8 votes
Accepted

How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
user avatar
8 votes

Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
user avatar
  • 23.8k
8 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
user avatar
  • 33.8k
8 votes
Accepted

What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
user avatar
  • 221
8 votes
Accepted

Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
user avatar
  • 33.8k
7 votes
Accepted

Can non-differentiable layer be used in a neural network, if it's not learned?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ...
user avatar
  • 23.8k
7 votes
Accepted

Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
user avatar
  • 33.8k
7 votes
Accepted

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is \begin{align} \frac{d}{...
user avatar
  • 33.8k
7 votes

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$ They are not just "equivalent". It is actually the exact same function, with two different ways to write ...
user avatar
  • 23.8k
7 votes
Accepted

Why is the perceptron criterion function differentiable?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
user avatar
6 votes
Accepted

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
user avatar
  • 33.8k
5 votes
Accepted

Deep Q-Learning: why don't we use mini-batches during experience reply?

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ...
user avatar
  • 23.8k
5 votes
Accepted

Do good approximations produce good gradients?

In general $|f(x) - f_k(x)| \leq \epsilon$ doesn't ensure $|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon$. And for neural networks there is no reason to believe it will happen either. You can also ...
user avatar
5 votes
Accepted

Does Musk know what gradient descent is?

I think Musk was using the terminology correctly though perhaps with hyperbole. I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a ...
user avatar
5 votes
Accepted

Eligibility vector for softmax policy with policy gradients

Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ...
user avatar
  • 2,226
5 votes

Is there an ideal range of learning rate which always gives a good result almost in all problems?

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the ...
user avatar
4 votes

Is the mean-squared error always convex in the context of neural networks?

Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ...
user avatar
  • 532
4 votes

What does the notation $\nabla_\theta \mathcal{L}$ mean?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element ...
user avatar
4 votes
Accepted

Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
user avatar
  • 204
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
user avatar
4 votes
Accepted

Are filters fixed or learned?

What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ...
user avatar
  • 33.8k
4 votes
Accepted

Can neuroevolution be combined with gradient descent?

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and ...
user avatar
  • 33.8k
4 votes
Accepted

Why evolutionary training of neural networks is not popular?

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers ...
user avatar
4 votes
Accepted

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
user avatar
4 votes
Accepted

What is relation between gradient descent and regularization in deep learning?

Usually, when talking about regularization for neural networks there are 3 main types: L1, L2 and dropout. All affect the gradient descent procedure. L1 and L2 regularization is implemented in the ...
user avatar
4 votes

Is there an ideal range of learning rate which always gives a good result almost in all problems?

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in ...
user avatar
4 votes

Why is gradient descent used over the conjugate gradient method?

When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
user avatar
  • 876
3 votes

What can be deduced about the "algorithm" of backpropagation/gradient descent?

There are 3 separate issues that are often confounded in Deep Learning and Neuroscience: Deep Learning is inspired by the way the biological brain works. Deep Learning is how the biological brain ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible