2
$\begingroup$

When applying multinomial Naive Bayes text classification, I get very small probabilities (around $10e^{-48}$), so there's no way for me to know which classes are valid predictions and which ones are not. I'd the probabilities to be in the interval $[0,1]$, so I can exclude classes in the prediction with say a score of 0.5 or less. How do I go about doing this?

This is what I've implemented:

$$c_{\operatorname{map}}=\underset{\operatorname{c \in C}}{\arg \max }(P(c \mid d))=\underset{\operatorname{c \in C}}{\arg \max }\left(P(c) \prod_{1 \leq k \leq n_d} P\left(t_{k} \mid c\right)\right)$$

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ I think your probabilities are split up among all possible outputs. If you add them all together, do they come out to 1? Order them by value and you have your predictions $\endgroup$
    – Zakk Diaz
    Jul 17, 2018 at 18:30
  • 1
    $\begingroup$ Use log probabilities instead to avoid underflow when dealing with naive bayes probability. instead of a product of probabilities, compute the argmax summation of log probabilities $\endgroup$
    – calveeen
    Apr 16, 2021 at 0:33

1 Answer 1

Reset to default
0
$\begingroup$

You could say that the class of the data (e.g. spam vs not spam) is a hidden quality that can be inferred through the observable features (e.g. message subject contains "bitcoin"),

$$P(C \; | \; F)$$

which says that the probability of the class $C$ is conditioned on the visibility of the feature $F$.

Using Bayes' theorem we can write

$$P(C \; | \; F) = \frac{P(F \; | \; C) \, P(C)}{P(F)}$$

Your problem is that you are not taking into account the evidence, $P(F)$, denominator in this expression.

Bayes' theorem label factors

You can have multiple features in your likelihood function

$$P(F \; | \; C) = P(F_1,F_2,F_3,\ldots \; | \; C)$$

and these can be independent

$$\prod_k P(F_k \; | \; C)$$

and then you have your prior

$$P(C) \prod_k P(F_k \; | \; C)$$

but what you are missing is the evidence factor in the denominator

$$P(C \; | \; F) = \frac{P(C) \prod_k P(F_k \; | \; C)}{\sum_C \scriptsize{P(C) \prod_k P(F_k \; | \; C)}}$$

This is the general view of the problem. Specifically, to fix your issue, evaluate the probabilities for all your classes, add them together, then divide each probability by that sum.

For instance, if you have two classes and the probabilities are $p_0$ and $p_1$, then write

$$P(C=0) = \frac{p_0}{p_0 + p_1}$$

and

$$P(C=1) = \frac{p_1}{p_0 + p_1}$$

which guarantees that

$$P(C=0) + P(C=1) = 1$$

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .