10 votes
Accepted

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
user avatar
  • 9,379
10 votes
Accepted

What loss function to use when labels are probabilities?

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$. You are right, ...
user avatar
8 votes

Can the mean squared error be negative?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it. A standard Mean Squared Error function cannot be ...
user avatar
  • 23.8k
7 votes
Accepted

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The derivative of $\mathcal{L_1}(y, x) = (\hat{y} - y)^2 = (f(x) - y)^2$ with respect to $\hat{y}$, where $f$ is the model and $\hat{y} = f(x)$ is the output of the model, is \begin{align} \frac{d}{...
user avatar
  • 33.8k
7 votes

Which function $(\hat{y} - y)^2$ or $(y - \hat{y})^2$ should I use to compute the gradient?

The MSE can be defined as $(\hat{y} - y)^2$, which should be equivalent to $(y - \hat{y})^2$ They are not just "equivalent". It is actually the exact same function, with two different ways to write ...
user avatar
  • 23.8k
6 votes
Accepted

What is the formula used to calculate the loss in the FaceNet model?

The loss function used is the triplet loss function. Let me explain it part by part. Notation The $f^a_i$ means the anchor input image. The $f^p_i$ means the <...
user avatar
  • 1,715
6 votes
Accepted

In which cases is the categorical cross-entropy better than the mean squared error?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real ...
user avatar
  • 33.8k
6 votes
Accepted

Why is the evidence equal to the KL divergence plus the loss?

In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\...
user avatar
  • 33.8k
6 votes

What is the reason we loop over epochs when training a neural network?

I am not expert on optimization, but I can share with you my knowledge of the topic. I think that the source of your confusion is that you assumed that, after the first epoch, we have reached a local ...
user avatar
  • 33.8k
5 votes
Accepted

Why L2 loss is more commonly used in Neural Networks than other loss functions?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE): MSE: L2 loss is continuously-differentiable across any domain, unlike L1 loss. This makes training more stable and allows ...
user avatar
4 votes
Accepted

What is an objective function?

The "objective function" is the function that you want to minimise or maximise in your problem. The expression "objective function" is used in several different contexts (e.g. machine learning or ...
user avatar
  • 33.8k
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
user avatar
4 votes
Accepted

Can gradient descent training be used for non-smooth loss functions?

Gradient descent and stochastic gradient descent can be applied to any differentiable loss function irrespective of whether it is convex or non-convex. The "differentiable" requirement ensures that ...
user avatar
  • 204
4 votes
Accepted

What are the major differences between cost, loss, error, fitness, utility, objective, criterion functions?

They are not all interchangeable. However, all these expressions are related to each other and to the concept of optimization. Some of them are synonymous, but keep in mind that these terms may not be ...
user avatar
  • 33.8k
4 votes
Accepted

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the ...
user avatar
4 votes

Loss function for choosing a subset of objects

The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a ...
user avatar
4 votes
Accepted

Why is the mean used to compute the expectation in the GAN loss?

It seems your question is concerned with how an empirical mean works. It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\...
user avatar
4 votes
Accepted

What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
user avatar
  • 1,230
4 votes
Accepted

Where is the mistake in my derivation of the GAN loss function?

I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
user avatar
3 votes
Accepted

How to obtain a formula for loss, when given an iterative update rule in gradient descent?

You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: https://github.com/JamesChuanggg/pytorch-REINFORCE/. First of all, I would like ...
user avatar
  • 33.8k
3 votes
Accepted

How can the sum of squared errors have negative gradient if it's defined as the squared of the error?

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result? In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...
user avatar
  • 23.8k
3 votes
Accepted

Why is the "square error function" sometimes defined with the constant 1/2 and sometimes with the constant 1/m?

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to: $$E_{...
user avatar
  • 9,379
3 votes

How can we process the data from both the true distribution and the generator?

Let's start at the beginning. GANs are models that can learn to create data that is similar to the data that we give them. When training a generative model other than a GAN, the easiest loss function ...
user avatar
  • 1,146
3 votes
Accepted

Is it possible with stochastic gradient descent for the error to increase?

Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot ...
user avatar
3 votes
Accepted

Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?

Lets start with question 1) how does JS-divergence handles zeros? by definition: \begin{align} D_{JS}(p||q) &= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\ &= \frac{...
user avatar
  • 2,249
3 votes

What's the difference between RMSE and Euclidean distance, and when to use a custom loss?

For the first question, RMSE and Euclidean distance have no difference, not that i know of. For the second question, you only need the common loss function for normal tasks. MSE is a common loss ...
user avatar
  • 1,715
3 votes

How do you interpret this learning curve?

Depends on what does 1 represent in your task. If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, ...
user avatar
  • 71
3 votes

How do you interpret this learning curve?

The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss ...
user avatar
  • 1,359
3 votes

What's the function that SGD takes to calculate the gradient?

Welcome to AI Stack exchange! You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the ...
user avatar
  • 1,344

Only top scored, non community-wiki answers of a minimum length are eligible