What is the advantage of having a stochastic/probabilistic classification procedure?

The classifiers I have encountered so far are as follows. Suppose we have two outcomes $A = \{0,1\}$. Given a feature vector $x$, we have calculated a probability for each outcome and return the outcome $a \in A$ for which the probability is highest.

Now, I encountered a classification procedure as follows: first, map each $x$ to a probability distribution on $A$ by a mapping $H$. To classify $x$, choose an outcome $a$ according to the distribution $H(x)$.

Why not use the deterministic classification? Suppose $H(x)$ is 1 with probability $0.75$. Then, the obvious choice for an outcome would be $1$ and not $0$.

  • $\begingroup$ Please, next time, edit your post to clarify it. $\endgroup$
    – nbro
    Jun 24 '20 at 23:52

There are multiple potential reasons for having stochastic predictions (instead of categorical/binary).

First, it often simplifies training and improves the training outcome when training a classifier on producing probabilities per class. For example, it allows for the usage of many nice loss functions like the Mean Squared Error (MSE), which is compatible with the famous back-prop algorithm. Moreover, improving classification based on the loss computed from stochastic probabilities per class allows for improving the classifier even further when the counting loss computed on predicted class labels has reached 0 already. So, if you only have binary decisions and your classifier gets all classes correct, training can stop immediately since the loss becomes 0. However, when predicting probabilities per class and iteratively driving the probability of the correct class towards 1, training can continue for much longer, even after all classes have been classified correctly already. So, the classifier trained on making probabilistic predictions can in the end perform much better since training can progress for much longer, allowing for increasingly pronounced discrimination between classes with every update. A nice introduction is given in this Stanford lecture recording.

Second, it is often nice for a user to know how much certainty versus uncertainty there is in a classification. If one class has 80% probability, this is a pretty clear decision in favor of the 80%-probability-class. However, if you only get categorical class labels returned by your classifier, you cannot possibly know whether the evidence in favor of the winning class was only marginally higher than that of the other class (as measured by the classifier) or whether there was a clear difference.

  • $\begingroup$ Great answer, thank you. $\endgroup$
    – MMM
    Jun 24 '20 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.