0
$\begingroup$

I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). Can I do this with MSE loss, or do I need to switch to a more complicated categorical cross-entropy loss function...?

$\endgroup$
3
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Sep 9, 2022 at 22:20
  • $\begingroup$ You sound confused; SGD is not a loss, it is an optimization method that can be used for minimizing various functions usually used as loss ones in NN (both for classification and regression). $\endgroup$
    – desertnaut
    Sep 10, 2022 at 0:02
  • $\begingroup$ Yeah sorry I meant to say MSE. $\endgroup$
    – Gamaray
    Sep 10, 2022 at 8:52

1 Answer 1

1
$\begingroup$

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.

A theoretical reason

Both aforementioned losses are negative logarithmic likelihoods for following parametric statistical models.

MSE corresponds to the parametric regression model $Y = f(\theta, X) + \epsilon$, where $\epsilon$ is a normally distributed zero-mean random error, independent of $X$ (and if you make $\epsilon$ not normally distributed, but Laplace-distributed, you will get another well known loss — mean average error).

Cross-entropy loss corresponds to a classification model with two classes $C_0$ and $C_1$ where $P(C_1|x) = f(\theta, x)$.

Minimizing those losses means getting a maximum likelihood estimator for the regression/classification parameter $\theta$.

A practical reason

Cross-entropy loss is just plain better in distinguishing “good” classification from a “bad” one.

Consider a following example:

Suppose we have $101$ object from class 1 and two experts deciding whether they are from class 0 or from class 1. The first expert made one blatant error, assigning score (probability of belonging to class 1) 0.00001 to one of the objects and 0.99 to the others. The second expert is always correct but a bit less confident — they assigned score 0.9 to all objects. Based on common sense, predictions of the second expert are obviously better than those of the first one. Now let’s look at the losses.

First expert has MSE $\sim 0.0099998$ and Cross-Entropy loss $\sim 12.52$.

Second expert has MSE $0.01$ and Cross-Entropy loss $\sim 10.64$.

$\endgroup$
1
  • $\begingroup$ +1 The same idea applies to Frank Harrell’s comments here. $\endgroup$
    – Dave
    Sep 10, 2022 at 18:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .