Intuitively, why can the training of a neural network be formulated as a probability estimation problem?

Question

Neural network training problems are oftentimes formulated as probability estimation problems (such as autoregressive models).

How does one intuitively understand this idea?

Answer 1

Consider the case of binary classification, i.e. you want to classify each input $x$ into one of two classes: $y_1$ or $y_2$. For example, in the context of object classification, $y_1$ could be "cat" and $y_2$ could be "dog", and $x$ is an image that contains one main object.

In certain cases, $x$ cannot be easily classified. For example, in object classification, if $x$ is a blurred image where there's some uncertainty about the object in the image, what should the output of the neural network be? Should it be $y_1$, $y_2$, or maybe it should be an uncertainty value (i.e. a probability) that lies between $y_1$ and $y_2$? The last option is probably the most reasonable, but also the most general one (in the sense that it can also be used in the case there's little or no uncertainty about what the object is).

That's the reason why we can model or formulate this (or other) supervised learning problem(s) as the estimation of a probability value (or probability distribution).

To be more concrete, you can formulate this binary classification problem as the estimation of the following probability

\begin{align} P(y_1 \mid x, \theta_i) \in [0, 1] \label{1}\tag{1} \end{align}

where $y_1$ is the first class (or label), $(x, y) \in \mathcal{D}$ is a labeled training example, where $y$ is the ground-truth label for the input $x$, $\theta_i$ are the parameters of the neural network at iteration $i$, so, intuitively, $P(y_1 \mid x, \theta_i) $ is a probability that represents how likely the neural network thinks that $x$ belongs to the class $y_1$ given the current estimate of the parameters. The probability that $x$ belongs to the other class is just $1 - P(y_1 \mid x, \theta_i) = P(y_2 \mid x, \theta_i)$. In this specific case, I have added a subscript to $\theta$ to indicate that this probability depends on the $i$th estimate of the parameters of the neural network.

Once you have $P(y_1 \mid x, \theta_i)$, if you want to perform classification, you will actually need to choose a threshold value $t$, such that, if $P(y_1 \mid x, \theta_i) > t$, then $x$ is classified as $y_1$, else it is classified as $y_2$. This threshold value $t$ can be $0.5$, but it can also not be.

Note that, in the case above, $P(y_1 \mid x, \theta_i)$ is a number and not a probability distribution. However, in certain cases, you can also formulate your supervised learning problem so that the output is a probability distribution (rather just a probability). There are also other problems where you don't estimate a conditional probability but maybe a joint probability, but the case above is probably the simplest one that should give you the intuition behind the idea of formulating machine learning problems as the estimation of probabilities or probability distributions.