I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: Bayesian Perceptron: Towards fully Bayesian Neural Networks.

I would love to understand this work, but, unfortunately, I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.

In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.

So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$. Because of the linearity, it can be exploited that also $\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$.

What I am confused about now is formula (14), which seems to show the compute the true posterior: $$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$

How is this formula of the posterior compatible with the Bayes Theorem? Where is the evidence, likelihood and prior?


Thanks for asking the question. I'm the author of the paper.

The key point is that the weights $w$ cannot be updated directly with the new data as $w$ is not directly related with the output $y$ (see equation (1)). First, it is necessary to update $a$ first, which actually is directly related with $y$ via the activation function, i.e., $y = f(a)$ with $f(.)$ being the activation function (see equation (2)). Hence, you will find the Bayes rule in equation (16) with all the quantities you are missing (likelihood, prior, etc.). This equation is used to calculate the posterior $p(a|D_i)$. This posterior is then used to update the weights by plugging $p(a|D_i)$ into equation (14). The solution of equation (14), i.e., the posterior mean and covariance matrix of $w$, is provided by means of equation (22). Algorithm 2 provides a summary of all necessary calculations.

I hope this answers your question. BTW: we have extended this algorithm to a full neural network. The scientific paper describing the whole procedure is currently under review.


You have two dependent variables $a$ and $w$. So, there is a joint distribution $p(w, a)$. You can make a marginalization by one of them, pretty much as you did in your second formula. $$p(w) = \int p(w, a)da$$ $$p(w) = \int p(w | a)p(a)da$$

The only difference in this case, the calculation made for the specific point $x_i, y_i$, which is empathized by sub-index on $p_i$ and conditioning on $D_i$

The key thing is that we can calculate the target distribution in many ways. With likelihood, evidence and prior you could indeed find posterior, but they are not always tractable/available. So, that's why in literature we usually differentiate true posterior and approximate posterior (or just posterior). Usually we get $p(w|d)$ with some form of approximation, but in the paper authors decide to get closed-form solution. That's why it was useful to represent in another way with intermediate distribution $a$. This would allow them to get closed-form for different activation functions.

So it's posterior in the sense of general context, not that specific formula, I think your limits of Bayesian knowledge is just fine

  • $\begingroup$ This doesn't seem to answer the main question. Yes, sure, you're explaining how marginalization works, but $p(w)$ typically denotes the marginal probability distribution (e.g. the prior) over $w$ and the posterior is typically denoted by $p(w \mid D)$, while the author of the paper really calls $p(w)$ the "true posterior distribution". How do you explain this? If that's a posterior, you can express it in terms of the likelihood, prior and evidence. I think that's the main question/doubt of the OP is. $\endgroup$ – nbro Mar 23 at 10:46
  • $\begingroup$ @m_3464gh Please, don't ask other questions in the comments if you already modified this post to ask that question. It's better you just ask the answerer to answer the other question by linking him to it. $\endgroup$ – nbro Mar 23 at 10:53
  • 1
    $\begingroup$ @nbro, yes thats exactly the main question here. $\endgroup$ – f_3464gh Mar 23 at 10:59
  • $\begingroup$ Thanks for answering @Kirill Fedyanin - With my knowledge of Bayesian statistics, the true posterior can always be calculated via likelihood prior / evidence. I can't see this in the referred paper. I also wonder why $a$ is used for marginalisation instead of $w$, because we want to get the posterior over the weights - but i already asked this question here if you have any idea on that: ai.stackexchange.com/questions/26950/… $\endgroup$ – f_3464gh Mar 23 at 11:00
  • $\begingroup$ @m_3464gh, I updated the answer $\endgroup$ – Kirill Fedyanin Mar 23 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.