Bayesian Perceptron: How is it compatible to Bayes Theorem?

Question

I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: Bayesian Perceptron: Towards fully Bayesian Neural Networks.

I would love to understand this work, but, unfortunately, I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.

In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.

So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$. Because of the linearity, it can be exploited that also $\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$.

What I am confused about now is formula (14), which seems to show the compute the true posterior: $$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$

How is this formula of the posterior compatible with the Bayes Theorem? Where is the evidence, likelihood and prior?

Answer 1

Thanks for asking the question. I'm the author of the paper.

The key point is that the weights $w$ cannot be updated directly with the new data as $w$ is not directly related with the output $y$ (see equation (1)). First, it is necessary to update $a$ first, which actually is directly related with $y$ via the activation function, i.e., $y = f(a)$ with $f(.)$ being the activation function (see equation (2)). Hence, you will find the Bayes rule in equation (16) with all the quantities you are missing (likelihood, prior, etc.). This equation is used to calculate the posterior $p(a|D_i)$. This posterior is then used to update the weights by plugging $p(a|D_i)$ into equation (14). The solution of equation (14), i.e., the posterior mean and covariance matrix of $w$, is provided by means of equation (22). Algorithm 2 provides a summary of all necessary calculations.

I hope this answers your question. BTW: we have extended this algorithm to a full neural network. The scientific paper describing the whole procedure is currently under review.

Answer 2

You have two dependent variables $a$ and $w$. So, there is a joint distribution $p(w, a)$. You can make a marginalization by one of them, pretty much as you did in your second formula. $$p(w) = \int p(w, a)da$$ $$p(w) = \int p(w | a)p(a)da$$

The only difference in this case, the calculation made for the specific point $x_i, y_i$, which is empathized by sub-index on $p_i$ and conditioning on $D_i$

The key thing is that we can calculate the target distribution in many ways. With likelihood, evidence and prior you could indeed find posterior, but they are not always tractable/available. So, that's why in literature we usually differentiate true posterior and approximate posterior (or just posterior). Usually we get $p(w|d)$ with some form of approximation, but in the paper authors decide to get closed-form solution. That's why it was useful to represent in another way with intermediate distribution $a$. This would allow them to get closed-form for different activation functions.

So it's posterior in the sense of general context, not that specific formula, I think your limits of Bayesian knowledge is just fine