1
$\begingroup$

In naive Bayes classification, we estimate the class of a document as follows

$$\hat{c} = \arg \max_{c \in C} P(c \mid d) = \arg \max_{c \in C} \dfrac{ P(d \mid c)P(c) }{P(d)} $$

It has been said in page 4 of this textbook that we can ignore the probability of document since it remains constant across classes.

We can conveniently simplify the above equation by dropping the denominator $p(d)$. This is possible because we will be computing $\dfrac{P(d \mid c)P(c)}{P(d)}$for each possible class. But $P(d)$ doesn't change for each class; we are always asking about the most likely class for the same document $d$, which must have the same probability $P(d)$. Thus, we can choose the class that maximizes this simpler formula

$$\hat{c} = \arg \max_{c \in C} P(c \mid d) = \arg \max_{c \in C} P(d \mid c)P(c) $$

Since the value of the document does not influence the choice of the class, naive Bayes algorithm does not consider that.

But, I want to know the value of $P(d)$. Is it $\dfrac{1}{N}$, if total number of documents are $N$? How should I calculate $P(d)$?

$\endgroup$
2
  • $\begingroup$ I suppose that the frequency of the document in the corpus would be the natural estimate of the probability of that document appearing in the corpus. So, generally, you could have the probability $M_d/N$, where $M_d$ is the number of times the specific document appears in the corpus. So, yeah, I would say that you're right, but I am not an expert on this topic, so I don't know what people are doing in this context (and that's also why I didn't post this as a formal answer). $\endgroup$
    – nbro
    Jul 6 at 1:53
  • $\begingroup$ The probability of "the cat sat on the mat" is high; the probability of "fghkjrthwerwerwec" is low. $\endgroup$
    – user253751
    Aug 6 at 11:25
0
$\begingroup$

$P(d)$ (aka evidence) is a probability of your data (observation) and is defined as follows:

$$ P(d) = \sum_i P(d|c_i)P(c_i) $$

for all classes $c$.

According to the book, $P(c)=\frac{N_c}{N_{doc}}$ and $P(d|c)$ is a likelihood and, applying the assumptions from the book, can be defined as $P(w_i |c)=\frac{\text{count}(w_i, c)+1}{\sum_{w \in V}\text{count}(w, c) + |V|}$, where $V$ consists of the union of all the word types in all classes.

Taking the example from Section 4.3 of the book,

Dataset Cat Documents
Training - just plain boring
- entirely predictable and lacks energy
- no surprises and very few laughs
+ very powerful
+ the most fun film of the summer
Test ? predictable with no fun

we'll get:

$$ P(-) = \frac{3}{5}\\ P(+) = \frac{2}{5}\\ P(S|−) = \frac{2\times 2\times 1}{34^3}\\ P(S|+) = \frac{1\times 1 \times 2}{29^3}\\ P(d) = P(S|−)P(−) + P(S|+)P(+) = 6.1\times 10^{−5} + 3.2\times 10^{−5} $$

$\endgroup$
5
  • $\begingroup$ Yes, this seems to make sense (as you're just applying marginalization), although I had not thought about it in this way, probably because I had not read that section of the book, so I didn't know whether the likelihood and evidence were given or how to compute them in this context. In any case, how would this be different from just computing the frequency of the document in the corpus (as the OP and I suggest in the comment above)? $\endgroup$
    – nbro
    Jul 6 at 11:00
  • $\begingroup$ @nbro The denominator is just a normalization constant and hence related to the likelihood times prior. Thus, the sum of the posterior for all categories is always equal to 1, which allows us to judge how confident we are in our predictions. $\endgroup$ Jul 6 at 11:20
  • $\begingroup$ Yes, I know that the denominator is a normalization constant, but you're not answering my question. So, if someone asked you to estimate the probability of a document, how would you estimate it? Why wouldn't you just compute the frequency of the document in the corpus? That's my question. Yes, sure, you can compute P(d) by marginalizing over $c$, but why wouldn't you simply use the frequency? Anyway, how do you get from $𝑃(w_i \mid c)$ to the likelihood? It may be something you should explain more in detail. $\endgroup$
    – nbro
    Jul 6 at 11:26
  • $\begingroup$ @nbro Bayes' rule is difficult to comprehend. I would argue we cannot consider all documents with the same probability. This would only be true if we had exact duplicates in the dataset, which doesn't make any sense. As for $𝑃(w_i \mid c)$, it is taken from Eq. 4.12 in the book, which we compute as the fraction of times each word $w_i$ of a sentence $S$ appears among all words in all documents of topic $c$. $\endgroup$ Jul 6 at 11:47
  • $\begingroup$ Bayes' rule is usually very easy to comprehend. Not sure what you mean by that. Of course, not all documents may have the same probability, but this is reflected in their potentially different frequency. Also not sure why you think that, generally, documents would have the same frequency. Finally, my question was: how do you relate $P(w_i \mid c)$ to $P(d \mid c)$, which is not explained in your answer. $\endgroup$
    – nbro
    Jul 6 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.