This is a theoretical question. Is it possible to overfit a model on infinite amounts of data?

Let me clarify there are no duplicates.

Say, we have a generator function that produces data, with the correct classification/regression value, and we can generate infinite amounts of valid data. How long does it take for the model to overfit?

This question arose because I'm training an RNN model for fake news classification, and MSE loss is almost always 0.000, only 25% of the training data.

Will it be possible to overfit with one epoch of training on the infinite data generator?

(I'm thinking what will happen is the model will either get perfect, or sync into the generator's non-perfect randomness, and learn nothing)

  • $\begingroup$ If your data covers the entire sample space (easy to conceptualize with discrete-valued features), you don't need to abstract any model at all - you can just spit out the mapping from input to output, which much exist in the training data. I'm not sure the concept of overfitting is meaningful if it's impossible to have "unseen data". There's also no concept of "noise", as every unique combination of feature inputs maps to one and only one output - if "noise" can be perfectly predicted from the data, it's signal. $\endgroup$ Jun 2 '21 at 16:32
  • $\begingroup$ This is to generalize the concept of which the dataset is large enough that the features learned at the beginning of the epoch practically don't exist by the end. This is currently the case with the model I am training. If we have perfect randomness included in the model, then the noise becomes impossible to predict. There is some 'unseen' data, although it is very arbitrary in state-space $\endgroup$ Jun 2 '21 at 17:31
  • $\begingroup$ I feel that there's a disconnect here between "very large data" and "infinite data". With an infinite sample size, you'll cover the entire sample space almost surely. When selecting a new test sample from the population, the probability that it's not already in an infinite training sample of that population is zero (no unseen data). I don't follow your point about noise - since there are no duplicates in the data, every input has a fixed, deterministic mapping to an output (there is no randomness, since the output is perfectly predictable from the input features). $\endgroup$ Jun 2 '21 at 18:00

The trivial answer is that yes, it is possible. Consider standard Gaussian data and a generator sampling points $1/n$ with $n \rightarrow \infty$. Since you never see points beyond $(0,1]$, you are unlikely to learn the parameters of the distribution.

More generally, and for a supervised problem with data $X,Y$, if your data generator covers a subset of the domain with probability $1-\delta$ (with respect to the distribution of $X,Y$) for a very small $\delta$ and your model $\hat{f}$ fits with error $\epsilon$ here, then your generalization error is $\mathbb{E}_{X Y}[\text{loss}(\hat{f}(X),Y)] \leq \epsilon + \delta \sup_{X Y} |\text{loss}(\hat{f}(X),Y)|$. So if you knew that the loss were bounded and you could make $\delta$ arbitrarily small, you wouldn't overfit, because there wouldn't be any region of the sample space significant enough where you wouldn't be able to generalize to.

In other words, it all boils down to your generator being able to cover a set with high enough probability and your loss being "well behaved".

Note that because you will only have a finite coverage of the domain, e.g. an $\epsilon$-cover, you need some assumption on the growth of the loss for points which are $\epsilon$ apart, e.g. a Lipschitz condition (and also for the learned $\hat{f}$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.