What's the likelihood in Bayesian Neural Networks?

Question

I'm trying to understand the concept behind BNN. Their are based on the Bayes Theorem:

$$p(w \mid \text{data}) = \frac{p(\text{data} \mid w)*p(w)}{p(\text{data})}$$

which boils down to

$$\text{posterior} = \frac{\text{likelihood} * \text{prior}}{\text{evidence}}.$$

I understand that if we assume a Gaussian distribution for the model, the likelihood function comprises a product (or sum if we use log) of each data point inserted into the Gaussian pdf. The parameters which we can change are $\mu$ and $\sigma^2$. We want to increase the likelihood because higher values of the Gaussian pdf means higher probability.

How do things work when using a neural network? I assume that the likelihood function looks s.th. like inserting each data point inside the calculation (weighted sums) of the neural network. But does the neural network need a softmax layer at the end so that we can interpret the outputs as probabilities/likelihoods? Or do we measure likelihood by applying some error measurement like cross-entropy or squared-loss?

Answer 1

The likelihood depends on the task that you are solving, so this is similar to traditional neural networks (in fact, even these neural networks have a probabilistic/Bayesian interpretation!).

• For binary classification, you should probably use a Bernoulli, which, in practice, corresponds to using a sigmoid with a binary cross-entropy (you can show that the minimization of the cross-entropy is equivalent to the maximization of Bernoulli p.m.f.)

• If it's a multi-class classification problem, you should use a categorical distribution, which corresponds to a softmax with a categorical cross-entropy; see e.g. this implementation

• If it's a regression problem, the likelihood could be a Gaussian (which is equivalent to using the MSE: you can also show this).

In the case of (mean-field) variational BNNs (VBNNs) (note that not all neural networks denoted as BNNs are VBNNs, but here I will only focus on VBNNs), rather than directly performing Bayesian inference (i.e. applying the Bayes theorem directly), you will perform approximate inference and, more specifically, you cast the inference problem as an equivalent optimization problem, where you typically optimize the loss function known as the evidence lower bound (ELBO), which is composed of two terms

1. the (log-)likelihood of the parameters (so the Bernoulli/categorical/Gaussian)
2. the KL divergence (the regularization part)

Theoretically, given that I am not a statistician, I cannot tell you right now why we can have a Bernoulli likelihood and a Gaussian prior (which is typically the case and, more specifically, in the case of this paper, they assume the weights to be independent, which is kind of a strong assumption but simplifies the computations, so this leads to a factorized multi-variate Gaussian over the weights), but this is what I had seen in practice.