0
$\begingroup$

I was playing around with some models and training them on a heart disease dataset and found my Extra Tree Classifier reported a 100% accuracy. I was shocked at first since I had never gotten a perfect score report from a model, but I realized that the model is most likely over/underfit. After doing some research, I found that you can determine if a model is over/underfit by comparing the Training Score and Cross Validation Score reports, but after looking at the report, I could not tell if my model was over/underfit or if it really is perfect. (I am new to machine learning and I have high doubts that it is actually perfect.) Here is the model report:

Extra Tree Classifier Model Report

Can someone help me figure out if this model is over/underfit? (If I need to give any more data/info to figure this out, please let me know!)

$\endgroup$
3
  • $\begingroup$ "you can determine if a model is over/underfit by comparing the Training Score and Cross Validation Score reports" > did you do that? How's the performance on the validation set? $\endgroup$
    – Martino
    Jul 6, 2022 at 8:16
  • $\begingroup$ Based off the report, the training score was 100 and the cross validation score was about 98. $\endgroup$ Jul 6, 2022 at 9:17
  • $\begingroup$ Turns out the model contained a whole lot of bias because 72% of the dataset was duplicated data. The reason for the CV and training score being so close is because the testing data was basically the same as the training data. $\endgroup$ Jul 8, 2022 at 3:35

1 Answer 1

0
$\begingroup$

To evaluate your fit, the results on the validation set, as you say, are essential.

  • A good performance on both the training and the validation set is a sign of a good fit
  • A poor performance on the training set (and therefore on the validations set) is a sign of underfitting
  • A good performance on the training set, coupled with a poor performance on the validation set is a sign of overfitting.

As you can see you cannot evaluate based on the training set alone, whether it's a good fit or an overfit of the training set.

However, you stated in a comment that you also got 98% on the validation set. We can therefore conclude that you do have a good fit and a working model.

The very high resulting score is probably due to the fact that you're working on a very easy task. The size shown in the image seems to be 100 samples per class -- this makes me think this is a 'toy' dataset used just as an exercise. Whether this works in the real world, whatever your objective is, depends on how representative this dataset is.

$\endgroup$
2
  • $\begingroup$ I see what you mean. The reason there's only 100 samples is because I split my dataset to be 80% for training and 20% for testing. Would increasing the testing split make it make it more obvious? I changed the split to be 50/50 just to see the difference. The new report has around 500 samples. It reported 100 training score and 93 cv score. $\endgroup$ Jul 6, 2022 at 18:05
  • $\begingroup$ The exact size doesn't matter as long as they are both statistically representative, and 80% training is kind of standard. But this isn't the point, what I mean is this is an "easy" task for the model to solve. $\endgroup$
    – Martino
    Jul 7, 2022 at 7:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .