1
$\begingroup$

Deep learning book chapter 6: In 6.2.1.2 last paragraph:

Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output units that saturate produce very small gradients when combined with these cost functions. This is one reason that the cross-entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution p(y | x).

explain the above sentence

Doubt: But we use mean squared error (MSE) and mean absolute error (MAE) for regression.

$\endgroup$
2
  • 1
    $\begingroup$ is there more context? i.e. are they talking about classification? I agree that in general the statement isn't true, MSE is used in regression as you point out, but for e.g. a classification task the statement would be correct. $\endgroup$
    – David
    Commented Nov 4, 2022 at 14:49
  • 2
    $\begingroup$ Yes, I went back through the sections and checked, it refers to classification, though not mentioned explicitly. $\endgroup$
    – vivian.ai
    Commented Nov 4, 2022 at 17:02

1 Answer 1

1
$\begingroup$

We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for regression, so why is it a problem for classification?

First, note that while this passage from the book does not explicitly specify whether it is talking about classification or regression, the mention of crossentropy loss necessitates that the discussion revolves around classification.

In linear regression, the outputs range is (-inf, inf), and in this case, the MSE and MAE functions are both convex and can be minimized. In classification, the output (assuming softmax activation of the final dense layer) is a vector of probabilities (with a probability for each class), each of which ranges from [0, 1] and together add to 1. In this situation, MSE and MAE are non-convex and may have a broad plateau. When you have "output units that saturate", you are in the plateau portion of the curve. In the plateau regions, gradients (slopes/derivatives) are very small and, therefore, training is very slow (see Fig 4 of article). Therefore, MSE and MAE "often lead to poor results when used with gradient-based optimization."

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .