# What is wrong in reasoning here in classification for defect detection?

Consider the following hypotheses:

$$H_0$$: a given coin is fair

$$H_1$$: a given coin is unfair

Let

$$\alpha$$ = P(Classify as $$H_1$$|Sample actually from $$H_0$$)

We know the statistics for a fair coin, which is that there is a 50–50 probability of the coin landing on heads or tails. As a result h, the number of heads, must be binomially distributed. Moreover, as N (the number of coin tosses) increases the Binomial distribution is increasingly well approximated as Gaussian.

Using this information, we can calculate the following for $$N=100$$ toses:

Here, I am confused. As the range for $$h$$ narrows around 50, that will mean that the coin is close to be a fair coin and hence the probability of classing it as unfair ($$\alpha$$) should decrease. However, the above table shows the opposite. What is wrong in my reasoning?

Alpha represents the significance level, or the probability that you will make a Type I error by rejecting the null hypothesis when it is actually true - in other words, the probability that you're wrong when you say the coin is unfair. This relates to how much evidence you need to claim that the coin is unfair - if you want to be very sure that the coin is indeed unfair, you need to see a big deviation from 50-50.

The ranges listed in the table show the "acceptable ranges" of observed outcomes at different alpha levels. As alpha gets smaller and the level of required evidence increases, we find that larger deviations from 50% are "acceptable" as statistical noise rather than unfairness of the coin. With small alpha, we need to be very sure we're not making a mistake when we call the coin unfair, so even moderate deviations from 50% aren't sufficient.

You could also think about it the other way, that the table shows you the alpha level you'd get when rejecting the null at a particular evidence level. If the coin shows heads 56% of the time and you call it unfair, you'll be wrong 20% of the time. If the coin shows heads 68% of the time and you call it unfair, you'll be wrong only 0.01% of the time.

For low alpha, you need to be very, very sure whenever you reject the null. To do, this you shouldn't reject the null very often, and should only reject it when you have lots of evidence. At an alpha of 0.0001, the observed outcome of the coin needs to differ from 50% by at least 18 percentage points before you call the coin significantly unfair. If the coin lands on heads 68% of the time, you can claim the coin is unfair and will only be wrong 0.01% of the time.

For high alpha, you're willing to reject the null more often and get it wrong more often. At an alpha of 0.2, you only need the outcome to differ from 50% by 6 percentage points before you call it significant. If the coin lands on heads 56% of the time, you can claim the coin is unfair but will be wrong 20% of the time.

• Thank you for the comment but it is not answer to my question. Your comment includes many things which are not defined (such as null hypothesis, type 1 error) and it is not an answer to why my reasoning is wrong. Sep 20, 2023 at 14:50
• @DSPinfinity The table shows the alpha you get by rejecting H0 at different levels of evidence. As the observed proportion $h$ narrows around 50% and you continue to call the coin significantly unfair, you are more likely to be wrong. Alpha isn't the probability of calling the coin unfair, it's the probability of incorrectly calling the coin unfair. If the experiment shows 56% heads and you call the coin unfair, you're much more likely to be wrong than if the experiment showed 68% heads an you called it unfair. Sep 20, 2023 at 15:04
• If you call the coin unfair anytime the experiment shows a result outside of 44%-56% heads, your claims that the coin is truly unfair will be wrong 20% of the time. If you call the coin unfair only when the experiment shows heads outside of 32%-68% of the time, you'll be wrong just 0.01% of the time. Sep 20, 2023 at 15:08
• I think your answer is still confusing and not clear for a non-expert. However, your first comment is convincing and very clear. As a result, it can be better if you use your first comment as an answer to the question. Sep 20, 2023 at 20:52