# How to calculate the confidence of a classifier's output?

I'm training a classifier and I want to collect incorrect outputs for human to double check.

the output of the classifier is a vector of probabilities for corresponding classes. for example, [0.9,0.05,0.05]

This means the probability for the current object being class A is 0.9, whereas for it being the class B is only 0.05 and 0.05 for C too.

In this situation, I think the result has a high confidence. As A's probability dominants B's and C's.

In another case, [0.4,0.45,0.15], the confidence should be low, as A and B are close.

What's the best formula to use to calculate this confidence?

• There's no best formula, this is heuristic based. It depends on what accuracy you're looking for, but if you want a place to start, consider anything > 0.85 in the correct class a confident prediction, anything between 0.3 and 0.85 low confidence, and anything beneath 0.3 wrong – Recessive Mar 3 '20 at 5:46
• If you have a well-calibrated method that does indeed output probabilities, your problem is already solved. In the first instance, the classifier says there's a 90% probability that the object belongs to class A, but in the second instance, it's only 45% sure that it belongs to class B. What more do you want? – Nuclear Hoagie Mar 3 '20 at 19:39