How to evaluate binary classifier on imbalanced dataset?

Question

I have trained a Decision Tree model on an imbalanced dataset. I got the following results for the test set from the sklearn and imblearn classification reports (attached below). Moreover, the other scores I calculated from sklearn are as follows:

AUROC (Area under Reciever operating curve) = 0.979

AUPR (Area under Precision recall curve) = 1.000

Recall = 0.99

Precision = 0.99

F1-score = 0.99

F2-score = 0.99

In my case, both classes are equally important. I have read that micro metrics (Prec, Recall, F1, F2, AUC) may not be good performance measures in imbalanced datasets as they can favor the majority class. For macro metrics, it is said that these are suited for multiclassification problems and may give a false narrative of the classifier in case we use them in binary classification. I am really confused if my classifier performing well or not. If it is performing well on which metrics should I justify this? Or how can I ensure my model is not biased?

Classification Report
               precision    recall  f1-score   support

           0       0.98      0.96      0.97       141
           1       1.00      1.00      1.00   1100416

    accuracy                           1.00   1100557
   macro avg       0.99      0.98      0.98   1100557
weighted avg       1.00      1.00      1.00   1100557

**************************************************************************************************

 Classification report imbalanced
                    pre       rec       spe        f1       geo       iba       sup

          0       0.98      0.96      1.00      0.97      0.98      0.95       141
          1       1.00      1.00      0.96      1.00      0.98      0.96   1100416

avg / total       1.00      1.00      0.96      1.00      0.98      0.96   1100557

```

Answer 1

To me your evaluation looks good. And I wouldn't say that all these metrics are not good for imbalanced problems. Especially F1 score and ROC are actually pretty good indicators for spotting biases in a model.

So unless you did a silly mistake like evaluating on training data you can say you trained a very good model and the metrics prove it. Now the question is if this model will generalize well also in a real use case scenario. I see that you have only 141 instances for the minority class in your test set. You don't say what type of data you're analyzing so we can't tell if the variability within that minority class is expected to be large or not. If you have reason to think it is, then you'll probably experience a drop in performance when using the model for real. Nothing in contrast with you're current evaluation, simply the results of testing on more representative data than your current test set.