In my code, I usually use the mean squared error (MSE), but the TensorFlow tutorials always use the categorical cross-entropy (CCE). Is the CCE loss function better than MSE? Or is it better only in certain cases?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number). However, in principle, you can use the MSE for classification problems too (even though that may not be a good idea). MSE can be preceded by the sigmoid function, which outputs a number $p \in [0, 1]$, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is $1 - p$.
Similarly, cross-entropy (CE) is mainly used for classification problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for binary and multi-class classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a softmax function (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid.
See also Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model? for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also How to choose cross-entropy loss in TensorFlow?, which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions? and Does the cross-entropy cost make sense in the context of regression?.
-
1$\begingroup$ tks, i have my thumb rule now, commonly, MSE for regression, CCE for classification $\endgroup$ – datdinhquoc Jan 20 '20 at 3:39
$\begingroup$
$\endgroup$
In a classification problem it's better to get higher error and higher error slope when we predict the label wrong.
As you see in the graph by using cross-entropy you get high error when the algorithm predict a label wrong and small error when the prediacted label is close enough, so it helps us to separate the predicted classes better.
-
1
