0
$\begingroup$

I'm using a Decision Tree that gave me great test metrics. Then I checked the learning curve, but it seems a little strange to me regarding the training score. Do you think there is a problem with overfitting or others problems? How can I solve it?

enter image description here

$\endgroup$
4
  • $\begingroup$ How exactly did you make this learning curve? $\endgroup$
    – Dr. Snoopy
    Jan 8, 2023 at 15:29
  • $\begingroup$ I used the LearningCurveDisplay function from sklearn $\endgroup$
    – giovasbr
    Jan 8, 2023 at 15:32
  • $\begingroup$ That learning curve is not used to detect overfitting, only a curve of loss vs training time (epochs) an be used for this purpose. $\endgroup$
    – Dr. Snoopy
    Jan 8, 2023 at 15:40
  • $\begingroup$ If my training and test score are close to 1.0, with respect to fit_time/score_time, is it a possible overfitting? $\endgroup$
    – giovasbr
    Jan 8, 2023 at 17:30

1 Answer 1

1
$\begingroup$

Overfitting occurs when the trained model fails to generalize performance to an independent test set. To assess overfitting, a metric for assessing model performance is needed. This can be measures of loss (e.g. log loss, Gini index) or "accuracy" (e.g. accuracy, F1 score). When the losses are higher or the accuracy is lower on the test data, you may have overfitting. Overfitting can be assessed as a function of multiple factors, including the number of training examples (as you have here) or model hyperparameters (e.g., maximum tree depth, minimum number of examples in leaf nodes).

Your plot seems to show that with fewer (100) training examples, your model does not generalize as well to test data (lower accuracy compared to training data). This may be overfitting. As you increase the amount of data, the performance on the test data gradually approaches the training data.

Overfitting is more common in "small" datasets, and increasing the size of a dataset is a known method of reducing overfitting. Thus, your observations appear to be consistent with known phenomena.

Do you think there is a problem with overfitting or others problems? There may be overfitting with using less training data, but this seems to be addressed by adding more training data, as explained above. Data leakage is also a consideration, but you would have to apply your domain knowledge to assess for that.

$\endgroup$
6
  • $\begingroup$ Is this an answer generated by ChatGPT? Because it does not answer the question, and does not consider that this curve is not the one used to detect overfitting. $\endgroup$
    – Dr. Snoopy
    Jan 8, 2023 at 20:22
  • $\begingroup$ I generated this 100% of this response. The question in the post was asking, "Do you think there is a problem with overfitting or others problems?". The response is "There may be overfitting with using less training data, but this seems to be addressed by adding more training data, as explained above. Data leakage is also a consideration, but you would have to apply your domain knowledge to assess for that." $\endgroup$ Jan 8, 2023 at 20:44
  • $\begingroup$ This SO article addresses the issue of whether the plot can evaluate overfitting. Main quote from here is "A learning curve shows the validation and training score of an estimator for varying numbers of training samples. It is a tool to find out how much we benefit from adding more training data and whether the estimator suffers more from a variance error or a bias error." The bias-variance issue is at the heart of evaluating overfitting. $\endgroup$ Jan 8, 2023 at 20:46
  • $\begingroup$ I provide one more article of where these types of plots are used to assess overfitting. The main quote here is: "Since the training score is very accurate, this indicates low bias and high variance. So this model also begins overfitting the data because the cross-validation score is relatively lower and increases very slowly as the size of the training set increases." $\endgroup$ Jan 8, 2023 at 21:04
  • $\begingroup$ These articles are wrong, in a learning curve you train multiple models, overfitting is seen in a learning curve of loss vs training steps, not versus the size of the training set. Also to evaluate bias/variance trade-off, you do this in a plot against the number of parameters in your model. These are important differences that are easy to overlook. $\endgroup$
    – Dr. Snoopy
    Jan 8, 2023 at 23:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .