0
$\begingroup$

If a model overfit to the training data, why does it generalize poorly?

Consider the basic problem of a noisy 2d dataset where I am fitting polynomials. A good model would be a parabola and a line would be underfitting. As I increase the polynomial's power, I end up overfitting more and more to the data. However, why is the data unlikely to follow this overfitting behavior? The only knowledge I have is of this data. It would be reasonable to assume the better the model does on this data, the better it would do on testing dataset.

How do I increase the complexity of my model or training data such that accuracy can only be improved? For example if I have a given model that does generalizes well, how do I keep this good generalization while it fits better to the training data?

$\endgroup$
1
  • $\begingroup$ I think you're asking different questions here. The answer below answers to your question in the title. Please, consider removing your other questions from this post and ask them in separate posts, if you want specific answers to them. Thank you. $\endgroup$
    – nbro
    Jan 3 at 11:12

2 Answers 2

3
$\begingroup$

The issue is the generalisation. If your model fits perfectly to your training data, it then depends whether your training data is a good reflection of the actual data you will encounter 'in the wild'. If it was a perfect sample, and no other values ever occur, then there is no problem.

But, as is more likely, the training data is only a small and not 100% representative sample. Now your model fits this data perfectly, but previously unseen (noisy) data is different, and doesn't fit, because your model is too specific.

With classification you want to allow for some variability, because real-world data is never perfect. Your accuracy (of classifying items) might improve if your model is more forgiving (ie less specific).

$\endgroup$
4
  • $\begingroup$ But the model is fitting to everything, the noise and also the actual data that it should be learning. So why does the noise ruin the generalization? $\endgroup$ Feb 27 at 20:10
  • $\begingroup$ @JobHunter69 The noise will be random, and thus will mask the 'actual' (idealised) data. But you don't know what is noise and what is the data, so the noise will be treated as part of the data and reduce accuracy. With more data, this effect diminishes. $\endgroup$ Feb 28 at 10:56
  • $\begingroup$ If the model fits both to the noise and also the data, why does the noise mask the data? The model should still be able to learn the data $\endgroup$ Feb 28 at 15:45
  • $\begingroup$ The model cannot distinguish between the two. $\endgroup$ Feb 29 at 9:06
2
$\begingroup$

Note that if you know the function your data was generated from, you do not even need machine learning. For example, if you generated the data with $y = f(x) = x^2$, you already have the function to predict future $y$s given the $x$s, i.e. $f(x) = x^2$.

So, we usually apply machine learning when we don't know the target function, i.e. when you don't know the shape of some function you want to learn, although you can make assumptions about it, which may be wrong. For example, you could assume your function is a parabola $x^2$, but it could also be any other function that may have data in common with $x^2$. So, if your limited training data suggests that the function is a parabola, but in reality is not, you will learn a parabola (overfit), instead of learning the actual target function (i.e. generalise). It can be a big problem if the target function is actually very different from the function you learned, which may be the case if you used small or non-representative training dataset, because your model will give wrong predictions for unseen data.

So, you almost never want to overfit.

$\endgroup$
1
  • 2
    $\begingroup$ Oliver's answer is correct. I'm just giving a slightly different answer. $\endgroup$
    – nbro
    Jan 3 at 11:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .