In Goodfellow et al. book Deep Learning chapter 12.1.4 they write

These large models learn some function $f(x)$, but do so using many more parameters than are necessary for the task. Their size is necessary only due to the limited number of training examples.

I am not able to understand this. Large models are expressive, but if you train them on few examples they should also overfit.

So, what do the authors mean by saying large models are necessary precisely because of the limited number of training examples?

This seems to go against the spirit of using more bias when training data is limited.


1 Answer 1


If you read the relevant section. it also says:

Model compression is applicable when the size of the original model is driven primarily by a need to prevent overfitting. In most cases, the model with the lowest generalization error is an ensemble of several independently trained models. Evaluating all $n$ ensemble members is expensive. Sometimes, even a single model generalizes better if it is large (for example, if it is regularized with dropout).

The keyword (I think) here is dropout. Dropout Learning in the referred book has been intepreted as training an ensemble of models, with a model probability same as the probability of a particular dropout architecture of the large Neural Network. Thus, this effectively makes the training as training multiple smaller Neural Nets. According to this paper on dropout, by the original authors, dropout prevents co-adaptation which effectively means you are just training an ensemble of Neural Nets. But this intuition lacks any theoretical justification.

Another paper (understanding the paper might require familiarity with certain statistical ideas of ML) claims this is not true, and dropout doesn't reduce co-adaptation but more likely reduces the variance over dropout patterns. They have provided better empirical and theoretical justifications to this end. So it is still up for debate what actually happens.

But in general the generalization error upper bound very roughly is directly proportional to the size of the Neural Nets. So yes the authors statement in face value is oversimplified and most likely wrong in the general case.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .