How can supervised learning be viewed as a conditional probability of the labels given the inputs?

In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,

$$\rho(\vec{y}|\vec{x},\vec{\theta})$$

where $$\vec{\theta}$$ denotes a learned set of network parameters, $$\vec{x}$$ is an arbitrary input, and $$\vec{y}$$ is an arbitrary output. If we assume we have already learned $$\vec{\theta}$$, then, in words, $$\rho(\vec{y}|\vec{x},\vec{\theta})$$ is the probability that the network will output an arbitrary $$\vec{y}$$ given an arbitrary input $$\vec{x}$$.

I am having a hard time reconciling how, after learning $$\vec{\theta}$$, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input $$\vec{x}$$, a trained network will always produce the same output.

Any insight would be appreciated.

• I agree, personally. I tend to describe it as a transformation operation from X to Y via Theta. Once the network is trained, the ultimate transformation is effectively deterministic unless you've created some type of random feedback layer somewhere. The transformation returns a probability or likelihood (as you certainly and clearly know) Jan 25, 2020 at 0:29

This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input $$x$$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.

People often use this notation to indicate that, in the case of classification, there is a categorical distribution over the labels given the inputs, but this can be misleading, as the softmax (the function often used to model this categorical distribution) only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network, although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a point estimate for each parameter of the network is learned and no uncertainty is properly modeled.

Nevertheless, certain supervised learning problems have a formal probabilistic interpretation. For example, the minimization of the mean squared error function is equivalent to the maximization of a log probability, assuming your probability distribution is a Gaussian with a mean equal to the output of your model. In this probabilistic interpretation, you typically attempt to learn a probability (e.g. of the labels in the training dataset) and not a probability distribution. Watch Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning) by G. Hinton or read the paper Bayesian Learning via Stochastic Dynamics or Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method by R. Neal for more details.

Moreover, there are Bayesian neural networks (BNNs), which actually maintain a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each parameter of the neural network, then you will learn the mean and variance of these Gaussians.

• Thank you for the insight! Is it even possible to interpret a (non-Bayesian) trained NN as a distribution over inputs/outputs? The only way I can think of to do so is to represent the distribution as a sum over delta functions, e.g., if the trained model should have input/outputs pairs of (x1, y1), (x2, y2), ..., then the trained distribution would be $p(y|x) = y1*\delta_x1 + y2*\delta_x2 + ...$ . If one applies this interpretation, then is there a relationship between the BNN distribution and the non-BNN one, e.g., the non-BNN distribution results from an MLE over the BNN one? Jan 25, 2020 at 17:47
• @Jammy I am not sure what you mean by a sum of delta functions to represent a probability distribution and how exactly does that yield a distribution. I've never heard anything similar, but I am not a statistician.
– nbro
Jan 25, 2020 at 18:29
• @Jammy You may want to try asking this question on Math SE or Cross Validated SE. I am also curious now to see a formal definition. What would be a distribution from which you always sample the same point?
– nbro
Jan 25, 2020 at 18:39