# How can the probability of two disjoint events be non-zero?

Let $$A$$ and $$B$$ be two models for a classification task. Let $$x$$ be a test set and $$M$$ be a metric for the classification task. $$X$$ be a random variable on test sets.

Now,

$$M(A,x) =$$ Score of model $$A$$ on test set $$x$$

$$M(B,x)$$ = Score of model $$B$$ on test set $$x$$

$$\delta(x) =$$ difference in performance of models wrt test set $$x$$ $$= M(A, x)-M(B,x)$$

Now, consider the following (statistical) hypothesis on the performance difference $$\delta$$

$$H_o : \delta(x) \le 0$$ $$H_1 : \delta(x) > 0$$

We define $$p-$$value as follows

$$P(\delta(X) \ge \delta(x) | H_o \text{is true} )$$

With this as context, I confused with the following paragraph (taken from p15 of Naive Bayes and Sentiment Classification)

So in our example, this $$p-$$value is the probability that we would see $$\delta(x)$$ assuming $$A$$ is not better than B. If $$\delta(x)$$ is huge (let’s say $$A$$ has a very respectable $$M$$ of $$.9$$ and $$B$$ has a terrible $$M$$ of only $$.2$$ on $$x$$), we might be surprised, since that would be extremely unlikely to occur if $$H_0$$ were in fact true, and so the $$p-$$value would be low (unlikely to have such a large $$\delta$$ if $$A$$ is in fact not better than $$B$$). But if $$\delta(x)$$ is very small, it might be less surprising to us even if $$H_0$$ were true and $$A$$ is not really better than $$B$$, and so the $$p-$$value would be higher.

It is told in the paragraph that $$p-$$value is very low if $$A's$$ performance is better than $$B$$.

I am thinking that $$p-$$value should be zero if $$A's$$ performance is better than $$B$$ since it is a disjoint event wrt $$H_0$$. Where am I going wrong?

• I don't have a definitive answer and I am also not a statistician, but from my understanding of statistical tests, the p-value is a probability of your test statistic, which can, for example, be the t-statistic (for the t-test). Therefore, the p-value depends on the probability distribution of the associated test statistic, which depends on your assumptions about our data (the t-statistic is computed as a function of your data). I think to fully answer this question it's required to understand what the test statistic and the associated prob. distribution are here.
– nbro
Commented Jul 7, 2021 at 10:15
• This is the type of question to which you should find more qualified people to answer on Cross Validated SE, as this is really a question about a statistics topic, although statistical tests are also used in AI and research areas that use or apply AI. I think we can consider this question on-topic here, as it's in the context of Naive Bayes and NLP, but it's something that we may want to discuss on meta.
– nbro
Commented Jul 7, 2021 at 10:19