What does that mean if my precision, F1-score are very high, but my ROC AUC score is around 0.5?

Question

What does it mean when my precision and so on are so high and the roc auc score is around 0.5?

Answer 1

If you are certain that your calculations are correct, then there may be class imbalance in your dataset. The metrics precision, f1_score, and accuracy are dominated by the model's performance on the major class in your dataset.

For example, if your dataset has $99$ examples of the positive class and $1$ example of the negative class and your model predicts positive $100$% of the time (no matter what), then

1. $Precision = \frac{TP}{(TP + FP)} = \frac{99}{(99 + 1)} = \frac{99}{100}$
2. $F_1 score = 2\times\frac{TP}{(2 \times TP + FP + FN)} = 2 \times \frac{99}{(2 \times 99 + 1 + 0)} = 198/199$
3. $Accuracy = \frac{(TP + TN)}{(TP + TN + FP + FN)} = \frac{(99 + 0)}{(99 + 0 + 1 + 0)} = \frac{99}{100}$

These give a false impression that your model is performing very well, but in fact, all it is doing is calling every example positive. This is why it is imperative to have a clear understanding of your performance metrics, what they mean, and what their limitations are. In general, it is also a good idea to examine multiple performance metrics, if appropriate.

In this example, specificity will reveal problems in the model and/or data.

1. $Specificity = \frac{TN}{(TN + FP)} = \frac{0}{(0 + 1)} = \frac{0}{1}$

Given all possible positive-negative example pairs in your dataset, the roc_auc is the proportion of those pairs for which the model correctly identifies the positive example. Thus, the AUC is more sensitive to class imbalance than those other metrics.

Your results also indicates that your model is not doing any better than the flip of a coin because an AUC of $0.5$ indicates the expected performance of a dummy (naïve) classifier.