I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes, however when looking at how $P(x|c_k)$ is calculated I don't understand it. The Wikipedia page says its calculated as follows $P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)} $.

They mention that $p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, does that mean $P(x|c_k)$? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is $p_{ki}$?

And in the first part, after the product symbol, are they raising this probability to the power pf $x_i$ or does that again just mean 'probability of class $c_k$ generating the term $x_i$'?

I also don't understand the intuition behind why or how this calculates $P(x|c_i)$.

Thanks for your help everyone.

  • $\begingroup$ Isn't $x$ a vector $[x_1, \dots, x_n]$? $\endgroup$ – nbro May 3 '20 at 16:10
  • $\begingroup$ @nbro Yes it's a feature vector although it can also be a matrix for larger data sets where the amount of rows is the 'number of samples', and the amount of columns is the 'number of features'. $\endgroup$ – Aguy May 3 '20 at 16:22
  • $\begingroup$ Therefore, that probability you're analysing can also be written as $P(x_1, \dots, x_n \mid c_k)$. Does this make things clearer? Now, what happens if you assume the observations are independent of each other? $\endgroup$ – nbro May 3 '20 at 16:23
  • $\begingroup$ @nbro I understand that. The problem I was facing is understanding what $p_{ki}$ is. And when in the formula they write $p_{ki}^{x_i}$ are they raising it to the power of $x_i$ or is it just saying that we're calculating $p_{ki}$ given $x_i$? $\endgroup$ – Aguy May 3 '20 at 16:26
  • $\begingroup$ It's been a while since I had to deal with naive Bayes. so I cannot help you more unless I read about it again (maybe I'll do that later), but have a look at the binomial distribution. The formulas look very similar. Anyway, I am pretty sure that this is an exponentiation operation, otherwise, what would the exponent $1 - x_i$ mean? In any case, $x_i$ must be encoded as a number. Maybe they mean $1 - i$ (rather than $1 - x_i$). Similarly, maybe they meant $p_{ki}^i$. $\endgroup$ – nbro May 3 '20 at 16:31

Bernoulli naïve Bayes

$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$

Let's examine the example of document classification.
Let K different text classes and n different terms that our vocabulary contains. $x_i$ are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document x. x is a vector of dimension n.

$P(x|c_k)$ is the probability that given the class k, document x to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x_i = 1$, then $1 - x_i = 0$ and inversely. In other words, for each term it takes the probability that the document does contain this term or it does not.

$p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, that is it could be the prior probability a document that belongs to k class contains this term of the vocabulary.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.