Are softmax outputs of classifiers true probabilities?

Question

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have mathematical properties of probabilities and are--in practice--presumed to be (conditional) probabilities of the classes given the features:

1. First, the softmax output for each class is between $0$ and $1$.
2. Second, the outputs of all the classes sum to $1$.

PROBLEM: However, just because they have mathematical properties of probabilities does not automatically mean that the softmax outputs are in fact probabilities. In fact, there are other functions that also have these mathematical properties, which are also occasionally used as activation functions.

QUESTION: "Do softmax outputs represent probabilities in the usual sense?" In other words, do they really reflect chances or likelihoods? (I use likelihood in the colloquial sense here.)

Answer 1

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, and another valid sense in which the answer is no.

No, they don't represent the probability

No, they do not represent the true probability.

You can think of a neural network as a function $f$. Let $f(y;x)$ denote the softmax output of the neural network corresponding to class $y$, on input $x$. Then $f(y;x)$ will typically not be equal to $p(y|x)$, the probability that sample $x$ is from class $y$. $f(y;x)$ can be viewed as an estimate of $p(y|x)$ -- as a best-effort guess at $p(y|x)$ -- but it can be an arbitrarily bad estimate/guess. Neural networks routinely make errors on tasks that even humans find clear. Also, neural networks have systematic biases. For instance, as the other answer explains, neural networks tend to be biased towards "overconfidence".

So you should not assume that the output from the neural network represents the true probability $p(y|x)$. There is some underlying probability. We might not know how to compute it, but it exists. Neural networks are an attempt to estimate it, but it is a highly imperfect estimate.

Yes, they do represent probabilities

While the softmax outputs are not the true probability $p(y|x)$, they do represent a probability distribution. You can think of them as an estimate of $p(y|x)$. For a number of reasons, it is an imperfect and flawed estimate, but it is an estimate nonetheless. (Even bad or noisy estimates are still estimates.)

Moreover, the way we train neural networks is designed to try to make them a good estimate -- or as good as possible. We train a neural network to minimize the expected loss. The expected loss is defined as

$$L = \mathbb{E}_x[H(p(y|x),f(y;x))],$$

where the expectation is with respect to $x$ chosen according to the data distribution embodied in the training set, and $H$ is the cross-entropy of the distribution $f(y;x)$ relative to the distribution $p(y|x)$. Intuitively, the smaller the training loss is, the closer that $f(y;x)$ is to $p(y|x)$.

So, neural networks are trained in a way that tries to make its output be as good an approximation to $p(y|x)$ as is possible, given the limitations of neural networks and given the training data that is available. As highlighted above, this is highly imperfect. But $f(y;x)$ does still represent a probability distribution, that is our attempt to estimate the true probability distribution $p(y|x)$.

Answer 2

Excellent question.

The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct.

What usually happens is that softmax probabilities for the predicted class are closer to 100% in all cases, whether the predictions are correct or incorrect, which effectively does not give you any information. This is called overconfidence.

This means that the probabilities are not useful, and you cannot really use them as reliable confidences to detect when the model is unsure or predicts incorrectly.

For reference: Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. In International conference on machine learning 2017 Jul 17 (pp. 1321-1330). PMLR.