How is a feed-forward neural network with few hidden layers and lots of nodes in those hidden layers different from a network with a lot of hidden layers but relatively lesser nodes in those hidden layers?
Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.
There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).
In practice, it is a very different story anyway:
- You need to train the network in an acceptable wall-clock time,
- able to fit stuff in memory
- You have limitations on maximum time for one inference
- You need to find the weights by a training algorithm
Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.
There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:
He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that skip connections are critical for really deep networks (> 20 learning layers).