In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,

$\ \ \ \ \ \rho(\vec{y}|\vec{x},\vec{\theta})$

where $\vec{\theta}$ denotes a learned set of network parameters, $\vec{x}$ is an arbitrary input, and $\vec{y}$ is an arbitrary output. If we assume we have already learned $\vec{\theta}$, then, in words, $\rho(\vec{y}|\vec{x},\vec{\theta})$ is the probability that the network will output an arbitrary $\vec{y}$ given an arbitrary input $\vec{x}$.

I am having a hard time reconciling how, after learning $\vec{\theta}$, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input $\vec{x}$, a trained network will always produce the same output.

Any insight would be appreciated.

  • $\begingroup$ I agree, personally. I tend to describe it as a transformation operation from X to Y via Theta. Once the network is trained, the ultimate transformation is effectively deterministic unless you've created some type of random feedback layer somewhere. The transformation returns a probability or likelihood (as you certainly and clearly know) $\endgroup$ – David Hoelzer Jan 25 at 0:29

This formulation can indeed be misleading, as the output of a neural network is usually deterministic (i.e. given the same input $x$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.

People often use this notation to indicate that there is a categorical distribution (in the case of classification) over the labels given the inputs, but this is misleading, as the softmax, the function often used to model this categorical distribution, only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network, although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a point estimate for each parameter of the network is learned and no uncertainty is properly modeled.

However, there are Bayesian neural networks (BNNs) and Bayesian deep learning, which actually have a formal probabilistic interpretation, that is, there's actually a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each weight of the net, then you will learn the mean and variance of these Gaussians.

Nevertheless, certain supervised learning problems have a probabilistic interpretation (I want to stress out the word "interpretation"). For example, minimizing the squared error is equivalent to maximizing a log probability assuming your probability distribution is a Gaussian with mean equal to the output of your model. See Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning) by G. Hinton for more details.

| improve this answer | |
  • $\begingroup$ Thank you for the insight! Is it even possible to interpret a (non-Bayesian) trained NN as a distribution over inputs/outputs? The only way I can think of to do so is to represent the distribution as a sum over delta functions, e.g., if the trained model should have input/outputs pairs of (x1, y1), (x2, y2), ..., then the trained distribution would be $p(y|x) = y1*\delta_x1 + y2*\delta_x2 + ...$ . If one applies this interpretation, then is there a relationship between the BNN distribution and the non-BNN one, e.g., the non-BNN distribution results from an MLE over the BNN one? $\endgroup$ – Jammy Jan 25 at 17:47
  • $\begingroup$ @Jammy I am not sure what you mean by a sum of delta functions to represent a probability distribution and how exactly does that yield a distribution. I've never heard anything similar, but I am not a statistician. $\endgroup$ – nbro Jan 25 at 18:29
  • $\begingroup$ @Jammy You may want to try asking this question on Math SE or Cross Validated SE. I am also curious now to see a formal definition. What would be a distribution from which you always sample the same point? $\endgroup$ – nbro Jan 25 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.