1
$\begingroup$

enter image description here

What does it mean when my precision and so on are so high and the roc auc score is around 0.5?

$\endgroup$
4
  • $\begingroup$ How many examples of each category do you have? $\endgroup$
    – Dave
    Nov 13, 2022 at 15:27
  • $\begingroup$ do you mean sample size? $\endgroup$ Nov 13, 2022 at 15:28
  • $\begingroup$ Broken down by category…I’m curious about how much imbalance you have in your data. $\endgroup$
    – Dave
    Nov 13, 2022 at 15:29
  • $\begingroup$ i think i did something wrong. sorry for the disturbance $\endgroup$ Nov 13, 2022 at 15:40

1 Answer 1

2
$\begingroup$

If you are certain that your calculations are correct, then there may be class imbalance in your dataset. The metrics precision, f1_score, and accuracy are dominated by the model's performance on the major class in your dataset.

For example, if your dataset has $99$ examples of the positive class and $1$ example of the negative class and your model predicts positive $100$% of the time (no matter what), then

  1. $Precision = \frac{TP}{(TP + FP)} = \frac{99}{(99 + 1)} = \frac{99}{100}$
  2. $F_1 score = 2\times\frac{TP}{(2 \times TP + FP + FN)} = 2 \times \frac{99}{(2 \times 99 + 1 + 0)} = 198/199$
  3. $Accuracy = \frac{(TP + TN)}{(TP + TN + FP + FN)} = \frac{(99 + 0)}{(99 + 0 + 1 + 0)} = \frac{99}{100}$

These give a false impression that your model is performing very well, but in fact, all it is doing is calling every example positive. This is why it is imperative to have a clear understanding of your performance metrics, what they mean, and what their limitations are. In general, it is also a good idea to examine multiple performance metrics, if appropriate.

In this example, specificity will reveal problems in the model and/or data.

  1. $Specificity = \frac{TN}{(TN + FP)} = \frac{0}{(0 + 1)} = \frac{0}{1}$

Given all possible positive-negative example pairs in your dataset, the roc_auc is the proportion of those pairs for which the model correctly identifies the positive example. Thus, the AUC is more sensitive to class imbalance than those other metrics.

Your results also indicates that your model is not doing any better than the flip of a coin because an AUC of $0.5$ indicates the expected performance of a dummy (naïve) classifier.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .