How is a feed-forward neural network with few hidden layers and lots of nodes in those hidden layers different from a network with a lot of hidden layers but relatively lesser nodes in those hidden layers?


Note: The following statements are about feed-forward neural networks. If you're interested in something else, please let me know.

There is a paper which proofs that one hidden layer is enough. So theoretically you could represent any network with many layers with a single hidden layer. I don't remember the title, though. Also I don't remember how it scaled (how many additional nodes you need per added layer).

In practice, it is a very different story anyway:

  1. You need to train the network in an acceptable wall-clock time,
  2. able to fit stuff in memory
  3. You have limitations on maximum time for one inference
  4. You need to find the weights by a training algorithm

Anecdotally, I can say wider networks are usually have more problems with overfitting / memory limitation problems while deeper networks have more problems with maximum inference time.

There is a nice paper from Microsoft where they trained the first super deep network. Many papers followed to that one:

He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

One main claim was that deeper networks learn more complex features. I don't remember how much of a problem overfitting was. Another important insight is that skip connections are critical for really deep networks (> 20 learning layers).

| improve this answer | |
  • $\begingroup$ "I don't remember the title, though", maybe, before posting this answer, you could have looked it up, as this answer, if good enough, could serve as reference? Anyway, you can still add this detail to your answer. I can say a similar argument regarding your "I don't remember how much of a problem overfitting was". I am asking you to cite and give the actual numbers because this is still an on-going area of research and many things are still poorly understood. Maybe this post may be useful to you: stats.stackexchange.com/q/182734/82135. $\endgroup$ – nbro Nov 12 '18 at 16:25
  • $\begingroup$ nbro, you seem to forget that I'm using my free time to answer this question. If you want me to put more effort in it, ask directly and politely for it. You leave a lot of comments here in this direction, that don't provide any value. $\endgroup$ – Martin Thoma Nov 12 '18 at 16:42
  • $\begingroup$ If you want to know how much overfitting is a problem for deep networks for a specific dataset in practice, I encourage you to try it. It's not hard, only time consuming. Please also don't forget to share the results if you do so. $\endgroup$ – Martin Thoma Nov 12 '18 at 16:47
  • $\begingroup$ I am also using my time to review questions and answers of this website, so that to make the website improve in quality. If you took the effort to write a 10-minute answer (or whatever), then I don't understand why would it be a problem to also look for a reference (1 or 2 minutes more). It's not about free time or not. It's about providing reliability or do nothing. $\endgroup$ – nbro Nov 12 '18 at 16:49
  • $\begingroup$ Then you should use your time more productive. Currently, for the comments I read here, I'd say you're a troll. From your six answers on this site I read two so far. One was good (+1), the other one has issues similar to what you criticize. You should first provide actual value before you make this site less attractive for active contributors. I showed you a way to do it. It's now in you. I'll also not answer to more contents here. If you want to chat, open a chat. $\endgroup$ – Martin Thoma Nov 12 '18 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.