Why MSE and MAE yield poor results when used with gradient-based optimization for classification?

Question

Deep learning book chapter 6: In 6.2.1.2 last paragraph:

Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output units that saturate produce very small gradients when combined with these cost functions. This is one reason that the cross-entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution p(y | x).

explain the above sentence

Doubt: But we use mean squared error (MSE) and mean absolute error (MAE) for regression.

Answer 1

We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for regression, so why is it a problem for classification?

First, note that while this passage from the book does not explicitly specify whether it is talking about classification or regression, the mention of crossentropy loss necessitates that the discussion revolves around classification.

In linear regression, the outputs range is (-inf, inf), and in this case, the MSE and MAE functions are both convex and can be minimized. In classification, the output (assuming softmax activation of the final dense layer) is a vector of probabilities (with a probability for each class), each of which ranges from [0, 1] and together add to 1. In this situation, MSE and MAE are non-convex and may have a broad plateau. When you have "output units that saturate", you are in the plateau portion of the curve. In the plateau regions, gradients (slopes/derivatives) are very small and, therefore, training is very slow (see Fig 4 of article). Therefore, MSE and MAE "often lead to poor results when used with gradient-based optimization."