# Tag Info

Accepted

### Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
• 10.4k

### Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
• 32.9k
Accepted

### What loss function to use when labels are probabilities?

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$. You are right, ...
• 2,056
Accepted

### What is an objective function?

The "objective function" is the function that you want to minimise or maximise in your problem. The expression "objective function" is used in several different contexts (e.g. machine learning or ...
• 41.1k
Accepted

### In which cases is the categorical cross-entropy better than the mean squared error?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real ...
• 41.1k
Accepted

• 1,660
Accepted

### How can we process the data from both the true distribution and the generator?

The Focus of This Question "How can ... we process the data from the true distribution and the data from the generative model in the same iteration? Analyzing the Foundational Publication In the ...
• 7,513
Accepted

### Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is \begin{align} \frac{d}{...
• 41.1k

### Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$ They are not just "equivalent". It is actually the exact same function, with two different ways to write ...
• 32.9k
Accepted

### What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
• 1,748
Accepted

### What is the formula used to calculate the loss in the FaceNet model?

The loss function used is the triplet loss function. Let me explain it part by part. Notation The $f^a_i$ means the anchor input image. The $f^p_i$ means the <...
• 1,745
Accepted

### In variational autoencoders, why do people use MSE for the loss?

If $p(x|z) \sim \mathcal{N}(f(z), I)$, then \begin{align} \log\ p(x|z) &\sim \log\ \exp(-(x-f(z))^2) \\ &\sim -(x-f(z))^2 \\ &= -(x-\hat{x})^2, \end{align} where $\hat{x}$, the ...
• 229

### What is the reason we loop over epochs when training a neural network?

I am not expert on optimization, but I can share with you my knowledge of the topic. I think that the source of your confusion is that you assumed that, after the first epoch, we have reached a local ...
• 41.1k
Accepted

• 2,990
Accepted

### What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?

They are not all interchangeable. However, all these expressions are related to each other and to the concept of optimization. Some of them are synonymous, but keep in mind that these terms may not be ...
• 41.1k
Accepted

### Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
• 191
Accepted

### Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
• 214
Accepted

### Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
Accepted

### Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?

Lets start with question 1) how does JS-divergence handles zeros? by definition: \begin{align} D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\ &= \frac{...
• 2,379
Accepted

### Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
• 224

### Loss function for choosing a subset of objects

The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a ...
• 5,378
Accepted

• 32.9k

### How can we process the data from both the true distribution and the generator?

Let's start at the beginning. GANs are models that can learn to create data that is similar to the data that we give them. When training a generative model other than a GAN, the easiest loss function ...
• 1,186
Accepted

### Analysis of Training Loss and Validation Loss Graph

Simply model 1 is a better fit compared to model 2. Graph for model 1 We notice that the training loss and validation loss ...