# Tag Info

Accepted

### Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...
• 26.5k

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
• 37k
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
• 37k

### Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
• 26.5k
Accepted

### Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
• 37k
Accepted

### What is the formula for the momentum and Adam optimisers?

I'm going to use slightly different notation, $\leftarrow$ for an assignment, $\alpha$ for learning rate, $\nabla_w J$ in place of $g$* and implied multiplication as these are slightly more common. ...
• 26.5k
Accepted

### What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
• 241
Accepted

### Can non-differentiable layer be used in a neural network, if it's not learned?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ...
• 26.5k
Accepted

### How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
• 7,176
Accepted

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\... • 37k 7 votes Accepted ### Which function$(\hat{y} - y)^2$or$(y - \hat{y})^2$should I use to compute the gradient? The derivative of$\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$with respect to$\hat{y}$, where$f$is the model and$\hat{y} = f(x)is the output of the model, is \begin{align} \frac{d}{... • 37k 7 votes ### Which function(\hat{y} - y)^2$or$(y - \hat{y})^2$should I use to compute the gradient? The MSE can be defined as$(\hat{y} - y)^2$, which should be equivalent to$(y - \hat{y})^2$They are not just "equivalent". It is actually the exact same function, with two different ways to write ... • 26.5k 7 votes Accepted ### Why is the perceptron criterion function differentiable?$\max(-y_i(w x_i), 0)$is not partial derivable respect$w$if$w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ... • 1,282 6 votes Accepted ### Are filters fixed or learned? What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ... • 37k 6 votes ### Is there an ideal range of learning rate which always gives a good result almost in all problems? The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the ... 6 votes ### Why is gradient descent used over the conjugate gradient method? When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ... • 1,131 5 votes Accepted ### Does Musk know what gradient descent is? I think Musk was using the terminology correctly though perhaps with hyperbole. I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a ... • 617 5 votes ### Is the mean-squared error always convex in the context of neural networks? Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ... • 562 5 votes Accepted ### Deep Q-Learning: why don't we use mini-batches during experience reply? DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ... • 26.5k 5 votes Accepted ### Do good approximations produce good gradients? In general$|f(x) - f_k(x)| \leq \epsilon$doesn't ensure$|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon. And for neural networks there is no reason to believe it will happen either. You can also ... • 200 5 votes Accepted ### Eligibility vector for softmax policy with policy gradients Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ... • 2,286 4 votes ### What does the notation\nabla_\theta \mathcal{L}\$ mean?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element ...
• 933
Accepted

### Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
• 204
Accepted

### Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
Accepted

### Can neuroevolution be combined with gradient descent?

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and ...
• 37k
Accepted

### Why evolutionary training of neural networks is not popular?

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers ...
• 9,037
Accepted

### Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
• 224
Accepted

### What is relation between gradient descent and regularization in deep learning?

Usually, when talking about regularization for neural networks there are 3 main types: L1, L2 and dropout. All affect the gradient descent procedure. L1 and L2 regularization is implemented in the ...