# Why do I get small probabilities when implementing a multinomial naive Bayes text classification model?

When applying multinomial Naive Bayes text classification, I get very small probabilities (around $$10e^{-48}$$), so there's no way for me to know which classes are valid predictions and which ones are not. I'd the probabilities to be in the interval $$[0,1]$$, so I can exclude classes in the prediction with say a score of 0.5 or less. How do I go about doing this?

This is what I've implemented:

$$c_{\operatorname{map}}=\underset{\operatorname{c \in C}}{\arg \max }(P(c \mid d))=\underset{\operatorname{c \in C}}{\arg \max }\left(P(c) \prod_{1 \leq k \leq n_d} P\left(t_{k} \mid c\right)\right)$$

• I think your probabilities are split up among all possible outputs. If you add them all together, do they come out to 1? Order them by value and you have your predictions – Zakk Diaz Jul 17 '18 at 18:30
• Zakk Diaz's comment should be an answer! – John Doucette Aug 15 '18 at 22:46

You could say that the class of the data (e.g. spam vs not spam) is a hidden quality that can be inferred through the observable features (e.g. message subject contains "bitcoin"),

$$P(C \; | \; F)$$

which says that the probability of the class $$C$$ is conditioned on the visibility of the feature $$F$$.

Using Bayes' theorem we can write

$$P(C \; | \; F) = \frac{P(F \; | \; C) \, P(C)}{P(F)}$$

Your problem is that you are not taking into account the evidence, $$P(F)$$, denominator in this expression. You can have multiple features in your likelihood function

$$P(F \; | \; C) = P(F_1,F_2,F_3,\ldots \; | \; C)$$

and these can be independent

$$\prod_k P(F_k \; | \; C)$$

and then you have your prior

$$P(C) \prod_k P(F_k \; | \; C)$$

but what you are missing is the evidence factor in the denominator

$$P(C \; | \; F) = \frac{P(C) \prod_k P(F_k \; | \; C)}{\sum_C \scriptsize{P(C) \prod_k P(F_k \; | \; C)}}$$

This is the general view of the problem. Specifically, to fix your issue, evaluate the probabilities for all your classes, add them together, then divide each probability by that sum.

For instance, if you have two classes and the probabilities are $$p_0$$ and $$p_1$$, then write

$$P(C=0) = \frac{p_0}{p_0 + p_1}$$

and

$$P(C=1) = \frac{p_1}{p_0 + p_1}$$

which guarantees that

$$P(C=0) + P(C=1) = 1$$