18
$\begingroup$

BACKGROUND: The softmax function is the most common choice for an activation function for the last dense layer of a multiclass neural network classifier. The outputs of the softmax function have mathematical properties of probabilities and are--in practice--presumed to be (conditional) probabilities of the classes given the features:

  1. First, the softmax output for each class is between $0$ and $1$.
  2. Second, the outputs of all the classes sum to $1$.

PROBLEM: However, just because they have mathematical properties of probabilities does not automatically mean that the softmax outputs are in fact probabilities. In fact, there are other functions that also have these mathematical properties, which are also occasionally used as activation functions.

QUESTION: "Do softmax outputs represent probabilities in the usual sense?" In other words, do they really reflect chances or likelihoods? (I use likelihood in the colloquial sense here.)

$\endgroup$
8
  • 1
    $\begingroup$ How would you define "represent probabilities"? What does that mean to you? What is the difference between "represent probabilities" vs "have the mathematical properties of probabilities"? $\endgroup$
    – D.W.
    Commented Nov 15, 2022 at 4:21
  • $\begingroup$ @D.W., by "represent probabilities", I mean "do they really reflect chances or likelihoods?" For example, if I toss a die lots of times, I expect each of the six faces to show up 1/6 of the time. Now let's assume we have a classification problem with 3 classes: A, B, and C.. If we sample lots of examples with a defined feature set, I might get a% are class A, b% are class B, and c% class C, empirically. Now, if I feed an example with that same feature set to a neural network, will the softmax probabilities be (a%, b%, c%) ? According to the answer below, the answer is no. $\endgroup$ Commented Nov 15, 2022 at 4:54
  • $\begingroup$ I do not call this overconfidence, this is called overconfidence in the literature, note that my answer is supported by references. The fact that they are not calibrated means they are not really probabilities (do not represent the true likelihood of the data). $\endgroup$
    – Dr. Snoopy
    Commented Nov 15, 2022 at 7:48
  • 1
    $\begingroup$ Anyways the answer you now accepted is plain wrong, not all estimates are useful. $\endgroup$
    – Dr. Snoopy
    Commented Nov 15, 2022 at 7:57
  • 2
    $\begingroup$ Regarding the issue of calibration, @D.W. points out that the issue of whether the softmax probabilities are "real probabilities" is nuanced. One the one hand, he presents a view that is in agreement with yours, but he also offers second perspective. In the end, I am able to hold both views in my mind comfortably. I thank you for taking the time to provide an excellent response and for providing the reference. $\endgroup$ Commented Nov 15, 2022 at 8:03

2 Answers 2

16
$\begingroup$

The answer is both yes, and no. Or, to put it another way, the answer depends on what exactly you mean by "represent probabilities", and there is a valid sense in which the answer is yes, and another valid sense in which the answer is no.

No, they don't represent the probability

No, they do not represent the true probability.

You can think of a neural network as a function $f$. Let $f(y;x)$ denote the softmax output of the neural network corresponding to class $y$, on input $x$. Then $f(y;x)$ will typically not be equal to $p(y|x)$, the probability that sample $x$ is from class $y$. $f(y;x)$ can be viewed as an estimate of $p(y|x)$ -- as a best-effort guess at $p(y|x)$ -- but it can be an arbitrarily bad estimate/guess. Neural networks routinely make errors on tasks that even humans find clear. Also, neural networks have systematic biases. For instance, as the other answer explains, neural networks tend to be biased towards "overconfidence".

So you should not assume that the output from the neural network represents the true probability $p(y|x)$. There is some underlying probability. We might not know how to compute it, but it exists. Neural networks are an attempt to estimate it, but it is a highly imperfect estimate.

Yes, they do represent probabilities

While the softmax outputs are not the true probability $p(y|x)$, they do represent a probability distribution. You can think of them as an estimate of $p(y|x)$. For a number of reasons, it is an imperfect and flawed estimate, but it is an estimate nonetheless. (Even bad or noisy estimates are still estimates.)

Moreover, the way we train neural networks is designed to try to make them a good estimate -- or as good as possible. We train a neural network to minimize the expected loss. The expected loss is defined as

$$L = \mathbb{E}_x[H(p(y|x),f(y;x))],$$

where the expectation is with respect to $x$ chosen according to the data distribution embodied in the training set, and $H$ is the cross-entropy of the distribution $f(y;x)$ relative to the distribution $p(y|x)$. Intuitively, the smaller the training loss is, the closer that $f(y;x)$ is to $p(y|x)$.

So, neural networks are trained in a way that tries to make its output be as good an approximation to $p(y|x)$ as is possible, given the limitations of neural networks and given the training data that is available. As highlighted above, this is highly imperfect. But $f(y;x)$ does still represent a probability distribution, that is our attempt to estimate the true probability distribution $p(y|x)$.

$\endgroup$
12
$\begingroup$

Excellent question.

The simple answer is no. Softmax actually produces uncalibrated probabilities. That is, they do not really represent the probability of a prediction being correct.

What usually happens is that softmax probabilities for the predicted class are closer to 100% in all cases, whether the predictions are correct or incorrect, which effectively does not give you any information. This is called overconfidence.

This means that the probabilities are not useful, and you cannot really use them as reliable confidences to detect when the model is unsure or predicts incorrectly.

For reference: Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. In International conference on machine learning 2017 Jul 17 (pp. 1321-1330). PMLR.

$\endgroup$
3
  • $\begingroup$ Related question: If I include several copies of each image in my training set, and label some copies to some classes, and other copies to other classes, should I expect the softmax output to be close to the distributions in the training set? For instance, if a particular image has 4 copies in the training set, with 3 copies in class A and 1 copy in class B, and I run the classifier on this image, should I expect the output on that image to be close to 75% for A and 25% for B? $\endgroup$
    – Stef
    Commented Nov 16, 2022 at 9:53
  • 1
    $\begingroup$ @Stef This will surely confuse the model but I do not think there is something generalizable about what would happen, it depends on the model, so you should not have an expectation. Label smoothing or mixup training would have the effect you mention. $\endgroup$
    – Dr. Snoopy
    Commented Nov 16, 2022 at 18:04
  • $\begingroup$ I'd like to point out that the original post contains 2 questions, in my view, and you're answering the second one correctly. First, we must say that, by the definition of a probability or probability vector, the softmax produces a probability vector and the sigmoid produces a probability (number between 0 and 1). Here, you answer the more relevant (in my view) question: do these mentioned probabilities represent good/true confidence/uncertainty values? $\endgroup$
    – nbro
    Commented Dec 14, 2022 at 18:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .