Assume we have a neural network and we want to train it on a classification problem. The hidden layers of the neural network are kind of feature representations of the input data.
If the neural network is big and the amount of data isn't enough for the complexity of the model, does it usually learn worse representations than a smaller neural network which is good for the amount of data we have?
Yes, it is a well known problem called Curse of Dimensionality. It happens when a finite number of data samples is used to train a network with a high-dimensional feature space (very deep network).
With regard to your question: yes, smaller networks (representational spaces with lower dimensions) describe better smaller datasets.
Why?
Because of data sparsity. As the features to represent a single sample increases the space between samples in the network representation also increases. When the samples are represented in too sparse space (all samples are very far away from each other in this high-dimensional space) then you can not draw any conclusions or relationships about them.
There is a balance between network dimensionality and dataset size, both of them must be in accordance to each other. You can think of this in this way: if data is represented in a very low dimensional space you can not tell them appart (1D space), if the data is represented in a very high dimensional space, then you can not find relationships within it. The dimensionality must be just the right one.
Image from this post
