1
$\begingroup$

I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes. However, when looking at how $P(x|c_k)$ is calculated, I don't understand it. The Wikipedia page says its calculated as follows

$$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}. $$

They mention that $p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, does that mean $P(x|c_k)$? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is $p_{ki}$?

And in the first part, after the product symbol, are they raising this probability to the power pf $x_i$ or does that again just mean 'probability of class $c_k$ generating the term $x_i$'?

I also don't understand the intuition behind why or how this calculates $P(x|c_i)$.

Improve this question
$\endgroup$
6
  • $\begingroup$ Isn't $x$ a vector $[x_1, \dots, x_n]$? $\endgroup$
    – nbro
    Commented May 3, 2020 at 16:10
  • $\begingroup$ @nbro Yes it's a feature vector although it can also be a matrix for larger data sets where the amount of rows is the 'number of samples', and the amount of columns is the 'number of features'. $\endgroup$
    – Aguy
    Commented May 3, 2020 at 16:22
  • $\begingroup$ Therefore, that probability you're analysing can also be written as $P(x_1, \dots, x_n \mid c_k)$. Does this make things clearer? Now, what happens if you assume the observations are independent of each other? $\endgroup$
    – nbro
    Commented May 3, 2020 at 16:23
  • $\begingroup$ @nbro I understand that. The problem I was facing is understanding what $p_{ki}$ is. And when in the formula they write $p_{ki}^{x_i}$ are they raising it to the power of $x_i$ or is it just saying that we're calculating $p_{ki}$ given $x_i$? $\endgroup$
    – Aguy
    Commented May 3, 2020 at 16:26
  • $\begingroup$ It's been a while since I had to deal with naive Bayes. so I cannot help you more unless I read about it again (maybe I'll do that later), but have a look at the binomial distribution. The formulas look very similar. Anyway, I am pretty sure that this is an exponentiation operation, otherwise, what would the exponent $1 - x_i$ mean? In any case, $x_i$ must be encoded as a number. Maybe they mean $1 - i$ (rather than $1 - x_i$). Similarly, maybe they meant $p_{ki}^i$. $\endgroup$
    – nbro
    Commented May 3, 2020 at 16:31

1 Answer 1

Reset to default
1
$\begingroup$

Bernoulli naïve Bayes

$P(x \mid c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$

Let's examine the example of document classification.
Let K different text classes and n different terms that our vocabulary contains. $x_i$ are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document $\mathbf{x}$. $\mathbf{x}$ is a vector of dimension $n$.

$P(x \mid c_k)$ is the probability that given the class $k$, document $\mathbf{x}$ to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x_i = 1$, then $1 - x_i = 0$ and inversely. In other words, for each term, it takes the probability that the document does contain this term or it does not.

$p_{ki}$ is the probability of class $c_k$ generating the term $x_i$, that is it could be the prior probability a document that belongs to $k$ class contains this term of the vocabulary.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .