I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). Can I do this with MSE loss, or do I need to switch to a more complicated categorical cross-entropy loss function...?
3
-
$\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$– Community BotCommented Sep 9, 2022 at 22:20
-
$\begingroup$ You sound confused; SGD is not a loss, it is an optimization method that can be used for minimizing various functions usually used as loss ones in NN (both for classification and regression). $\endgroup$– desertnautCommented Sep 10, 2022 at 0:02
-
$\begingroup$ Yeah sorry I meant to say MSE. $\endgroup$– GamarayCommented Sep 10, 2022 at 8:52
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.
A theoretical reason
Both aforementioned losses are negative logarithmic likelihoods for following parametric statistical models.
MSE corresponds to the parametric regression model $Y = f(\theta, X) + \epsilon$, where $\epsilon$ is a normally distributed zero-mean random error, independent of $X$ (and if you make $\epsilon$ not normally distributed, but Laplace-distributed, you will get another well known loss — mean average error).
Cross-entropy loss corresponds to a classification model with two classes $C_0$ and $C_1$ where $P(C_1|x) = f(\theta, x)$.
Minimizing those losses means getting a maximum likelihood estimator for the regression/classification parameter $\theta$.
A practical reason
Cross-entropy loss is just plain better in distinguishing “good” classification from a “bad” one.
Consider a following example:
Suppose we have $101$ object from class 1 and two experts deciding whether they are from class 0 or from class 1. The first expert made one blatant error, assigning score (probability of belonging to class 1) 0.00001 to one of the objects and 0.99 to the others. The second expert is always correct but a bit less confident — they assigned score 0.9 to all objects. Based on common sense, predictions of the second expert are obviously better than those of the first one. Now let’s look at the losses.
First expert has MSE $\sim 0.0099998$ and Cross-Entropy loss $\sim 12.52$.
Second expert has MSE $0.01$ and Cross-Entropy loss $\sim 10.64$.
