# How can supervised learning be viewed as a conditional probability of the labels given the inputs?

In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,

$$\rho(\vec{y}|\vec{x},\vec{\theta})$$

where $$\vec{\theta}$$ denotes a learned set of network parameters, $$\vec{x}$$ is an arbitrary input, and $$\vec{y}$$ is an arbitrary output. If we assume we have already learned $$\vec{\theta}$$, then, in words, $$\rho(\vec{y}|\vec{x},\vec{\theta})$$ is the probability that the network will output an arbitrary $$\vec{y}$$ given an arbitrary input $$\vec{x}$$.

I am having a hard time reconciling how, after learning $$\vec{\theta}$$, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input $$\vec{x}$$, a trained network will always produce the same output.

Any insight would be appreciated.

• I agree, personally. I tend to describe it as a transformation operation from X to Y via Theta. Once the network is trained, the ultimate transformation is effectively deterministic unless you've created some type of random feedback layer somewhere. The transformation returns a probability or likelihood (as you certainly and clearly know) – David Hoelzer Jan 25 at 0:29

This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input $$x$$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.
• Thank you for the insight! Is it even possible to interpret a (non-Bayesian) trained NN as a distribution over inputs/outputs? The only way I can think of to do so is to represent the distribution as a sum over delta functions, e.g., if the trained model should have input/outputs pairs of (x1, y1), (x2, y2), ..., then the trained distribution would be $p(y|x) = y1*\delta_x1 + y2*\delta_x2 + ...$ . If one applies this interpretation, then is there a relationship between the BNN distribution and the non-BNN one, e.g., the non-BNN distribution results from an MLE over the BNN one? – Jammy Jan 25 at 17:47