Let $A$ and $B$ be two models for a classification task. Let $x$ be a test set and $M$ be a metric for the classification task. $X$ be a random variable on test sets.


$M(A,x) = $ Score of model $A$ on test set $x$

$M(B,x)$ = Score of model $B$ on test set $x$

$\delta(x) =$ difference in performance of models wrt test set $x$ $= M(A, x)-M(B,x)$

Now, consider the following (statistical) hypothesis on the performance difference $\delta$

$$H_o : \delta(x) \le 0$$ $$H_1 : \delta(x) > 0$$

We define $p-$value as follows

$$P(\delta(X) \ge \delta(x) | H_o \text{is true} ) $$

With this as context, I confused with the following paragraph (taken from p15 of Naive Bayes and Sentiment Classification)

So in our example, this $p-$value is the probability that we would see $\delta(x)$ assuming $A$ is not better than B. If $\delta(x)$ is huge (let’s say $A$ has a very respectable $M$ of $.9$ and $B$ has a terrible $M$ of only $.2$ on $x$), we might be surprised, since that would be extremely unlikely to occur if $H_0$ were in fact true, and so the $p-$value would be low (unlikely to have such a large $\delta$ if $A$ is in fact not better than $B$). But if $\delta(x)$ is very small, it might be less surprising to us even if $H_0$ were true and $A$ is not really better than $B$, and so the $p-$value would be higher.

It is told in the paragraph that $p-$value is very low if $A's$ performance is better than $B$.

I am thinking that $p-$value should be zero if $A's$ performance is better than $B$ since it is a disjoint event wrt $H_0$. Where am I going wrong?

  • 1
    $\begingroup$ I don't have a definitive answer and I am also not a statistician, but from my understanding of statistical tests, the p-value is a probability of your test statistic, which can, for example, be the t-statistic (for the t-test). Therefore, the p-value depends on the probability distribution of the associated test statistic, which depends on your assumptions about our data (the t-statistic is computed as a function of your data). I think to fully answer this question it's required to understand what the test statistic and the associated prob. distribution are here. $\endgroup$
    – nbro
    Commented Jul 7, 2021 at 10:15
  • $\begingroup$ This is the type of question to which you should find more qualified people to answer on Cross Validated SE, as this is really a question about a statistics topic, although statistical tests are also used in AI and research areas that use or apply AI. I think we can consider this question on-topic here, as it's in the context of Naive Bayes and NLP, but it's something that we may want to discuss on meta. $\endgroup$
    – nbro
    Commented Jul 7, 2021 at 10:19


You must log in to answer this question.

Browse other questions tagged .