6 votes
Accepted

In which cases is the categorical cross-entropy better than the mean squared error?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real ...
  • 34.9k
6 votes
Accepted

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
  • 34.9k
3 votes

In which cases is the categorical cross-entropy better than the mean squared error?

In a classification problem it's better to get higher error and higher error slope when we predict the label wrong. As you see in the graph by using cross-entropy you get high error when the algorithm ...
  • 131
3 votes
Accepted

Why is the "square error function" sometimes defined with the constant 1/2 and sometimes with the constant 1/m?

The first variation is named "$E_{total}$". It contains a sum which is not very well-specified (has no index, no limits). Rewriting it using the notation of the second variation would lead to: $$E_{...
  • 9,519
2 votes

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

How are multiple local minima on the equation of a parabola possible, if a parabola has only one minimum? A parabola has one minimum, and no separate local minima. So it isn't possible. However... ...
  • 24.6k
2 votes

How to express accuracy of a regression ANN that uses MSE loss function?

You can not use error to reliably measure accuracy. Error is best used as a measure of how fast the model is currently learning. As an example, using different loss functions (cross entorpy vs MSE) ...
  • 1,266
2 votes

What does it mean if classification error is equal between two networks but the MSE is different?

Accuracy itself isn't a sufficient way to compare two models. For example, you need to consider the precision and recall stats (see confusion matrix) and calculate some other metrics like f1 score. ...
  • 121
2 votes
Accepted

What does it mean if classification error is equal between two networks but the MSE is different?

MSE just measures the squared difference between actual and target values. It can still correctly classify the values, but perhaps not with the same confidence - leading to a higher loss (e.g. an ...
2 votes
Accepted

How do I prove that the MSE is zero when all predictions are equal to the corresponding labels?

This is very easy to prove. Let's first prove that, if $\hat{y}_k = y_k$, then the $E = 0$. I will leave all steps, so that it's super clear. \begin{align} E &=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 ...
  • 34.9k
2 votes

What are the advantages of the Kullback-Leibler over the MSE/RMSE?

KL-divergence is a measure on probability distributions. It essentially captures the information loss between ground truth distribution and predicted. L2-norm/MSE/RMSE doesn't do well with ...
  • 1,369
2 votes

In variational autoencoders, why do people use MSE for the loss?

On page 5 of the VAE paper, it's clearly stated We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary ...
  • 34.9k
2 votes

In which cases is the categorical cross-entropy better than the mean squared error?

We sometimes see that binary cross-entropy (BCE) loss is used for regression problems. This post is my opinion on using BCE for regression problems. The figure below is the plots of BCE, $-t*\log(x) -...
1 vote
Accepted

In variational autoencoders, why do people use MSE for the loss?

If $p(x|z) \sim \mathcal{N}(f(z), I)$, then \begin{align} \log\ p(x|z) &\sim \log\ \exp(-(x-f(z))^2) \\ &\sim -(x-f(z))^2 \\ &= -(x-\hat{x})^2, \end{align} where $\hat{x}$, the ...
  • 139
1 vote
Accepted

What error should I use for RNN?

To provide a good answer would fill several pages. To keep it very simple try many different loss functions on your model. Your goal is to have the highest performance based on some desired ...
1 vote
Accepted

Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation. Mean square error for training data Given some training ...
  • 940
1 vote
Accepted

What is the meaning of these equations in Noise2Noise paper?

The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs ...
  • 600
1 vote
Accepted

What is the definition of a loss function in the context of neural networks?

A loss function is what helps you "train" your neural network to do what you want it to do. A better way to word it to begin with would be an "objective" function. This function ...
  • 156
1 vote

What is the advantage of using cross entropy loss & softmax?

Short answer: larger gradients That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to ...
  • 470
1 vote

Why do we calculate the mean squared error loss to improve the value approximation in Advantage Actor-Critic Algorithm?

I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to ...
1 vote

Is it normal to have the root mean squared error greater on the test dataset than on the training dataset?

It is common to have root mean squared error (RMSE) greater on the test dataset than on the training dataset (this is equal to having accuracy/score higher for model in training dataset than test ...
1 vote

How to determine the target value when using ReLU as activation function?

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good. Fist of all, with the only one layer and one neuron neural networks it does ...
  • 91
1 vote

How to determine the target value when using ReLU as activation function?

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some ...
  • 34.9k
1 vote

How to express accuracy of a regression ANN that uses MSE loss function?

Just as a general remark, notice that technically we don't use the term "accuracy" for regression settings, such as yours - only for classification ones. If RMSE is 'in the units of the quantity ...
1 vote
Accepted

What are the advantages of the Kullback-Leibler over the MSE/RMSE?

In the context of Variational Inference (VI): the KL allows you to move from the unknown posterior $p(z \mid x)$, to the known joint $p(z,x)=p(x|z)p(z)$ and optimize only the ELBO. You cannot do this ...
1 vote

Is it a good idea to train a CNN to detect the hydration value (percentage) in skin images and evaluate it with the MSE?

Your initial idea seems about right. Before creating your own classifier you might want to try transfer learning, using some pretrained network like VGG16 that is incorporated in most of machine ...

Only top scored, non community-wiki answers of a minimum length are eligible