# Tag Info

Accepted

### Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
Accepted

### What is the relationship between gradient accumulation and batch size?

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (...
Accepted

### Is neural networks training done one-by-one?

Should I be changing the weights/biases on every single sample before moving on to the next sample, You can do this, it is called stochastic gradient descent (SGD) and typically you will shuffle the ...

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...

### Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
Accepted

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\... 9 votes Accepted ### What is the formula for the momentum and Adam optimisers? I'm going to use slightly different notation,$\leftarrow$for an assignment,$\alpha$for learning rate,$\nabla_w J$in place of$g$* and implied multiplication as these are slightly more common. ... 8 votes Accepted ### Can non-differentiable layer be used in a neural network, if it's not learned? It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ... 8 votes Accepted ### How to avoid falling into the "local minima" trap? There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ... 7 votes Accepted ### Which function$(\hat{y} - y)^2$or$(y - \hat{y})^2$should I use to compute the gradient? The derivative of$\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$with respect to$\hat{y}$, where$f$is the model and$\hat{y} = f(x)is the output of the model, is \begin{align} \frac{d}{... 7 votes ### Which function(\hat{y} - y)^2$or$(y - \hat{y})^2$should I use to compute the gradient? The MSE can be defined as$(\hat{y} - y)^2$, which should be equivalent to$(y - \hat{y})^2$They are not just "equivalent". It is actually the exact same function, with two different ways to write ... 7 votes Accepted ### Why is the perceptron criterion function differentiable?$\max(-y_i(w x_i), 0)$is not partial derivable respect$w$if$w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ... 7 votes ### Why is gradient descent used over the conjugate gradient method? When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ... 6 votes Accepted ### Are filters fixed or learned? What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ... 6 votes ### Is there an ideal range of learning rate which always gives a good result almost in all problems? The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate. The paper's primary focus is the ... 5 votes Accepted ### Does Musk know what gradient descent is? I think Musk was using the terminology correctly though perhaps with hyperbole. I believe this was tweeted in the context of the botnet attacks on name-resolution services that broke Netflix and a ... 5 votes ### Is the mean-squared error always convex in the context of neural networks? Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ... 5 votes Accepted ### Do good approximations produce good gradients? In general$|f(x) - f_k(x)| \leq \epsilon$doesn't ensure$|\nabla f(x) - \nabla f_k(x)| \leq c\epsilon. And for neural networks there is no reason to believe it will happen either. You can also ... 5 votes Accepted ### Deep Q-Learning: why don't we use mini-batches during experience reply? DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ... 5 votes Accepted ### Why evolutionary training of neural networks is not popular? The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers ... 5 votes Accepted ### Eligibility vector for softmax policy with policy gradients Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ... 5 votes Accepted ### Combine multiple losses with gradient descent This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--... 4 votes ### What does the notation\nabla_\theta \mathcal{L}\$ mean?

This is standard backpropagation. The gradient term you see is in fact a vector of partial derivatives where each element is the partial derivative of the log-likelihood with respect to each element ...
Accepted

### Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
Accepted

### Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
Accepted

### Can neuroevolution be combined with gradient descent?

The paper The Comparison and Combination of Genetic and Gradient Descent Learning in Recurrent Neural Networks: An Application to Speech Phoneme Classification (2007), by Rohitash Chandra and ...
Accepted

### Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...