In Goodfellow et al. book Deep Learning chapter 12.1.4 they write

These large models learn some function $f(x)$, but do so using many more parameters than are necessary for the task. Their size is necessary only due to the limited number of training examples.

I am not able to understand this. Large models are expressive, but if you train them on few examples they should also overfit.

So, what do the authors mean by saying large models are necessary precisely because of the limited number of training examples?

This seems to go against the spirit of using more bias when training data is limited.

  • $\begingroup$ It's not the first time that I see someone complaining about something in that book, which turns out to be misleading/wrong. This is a very good example. In fact, "Their size is necessary only due to the limited number of training examples." is highly misleading if not completely wrong, given the previous sentence. $\endgroup$ – nbro Aug 23 at 11:16

If you read the relevant section. it also says:

Model compression is applicable when the size of the original model is driven primarily by a need to prevent overfitting. In most cases, the model with the lowest generalization error is an ensemble of several independently trained models. Evaluating all $n$ ensemble members is expensive. Sometimes, even a single model generalizes better if it is large (for example, if it is regularized with dropout).

The keyword (I think) here is dropout. Dropout Learning in the referred book has been intepreted as training an ensemble of models, with a model probability same as the probability of a particular dropout architecture of the large Neural Network. Thus, this effectively makes the training as training multiple smaller Neural Nets. According to this paper on dropout, by the original authors, dropout prevents co-adaptation which effectively means you are just training an ensemble of Neural Nets. But this intuition lacks any theoretical justification.

Another paper (understanding the paper might require familiarity with certain statistical ideas of ML) claims this is not true, and dropout doesn't reduce co-adaptation but more likely reduces the variance over dropout patterns. They have provided better empirical and theoretical justifications to this end. So it is still up for debate what actually happens.

But in general the generalization error upper bound very roughly is directly proportional to the size of the Neural Nets. So yes the authors statement in face value is oversimplified and most likely wrong in the general case.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.