In this note Justin Domke says that

In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.

But in Bengio's remark, he says

Very simple. Just keep adding layers until the test error does not improve anymore.

There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?

  • $\begingroup$ I would kindly request you to revise your question post here Cross Validated $\endgroup$ – quintumnia Aug 28 '17 at 10:50
  • $\begingroup$ just do trial and error with common sense until you get a natural hang of it - (my ML/NN prof) $\endgroup$ – k.c. sayz 'k.c sayz' Aug 28 '17 at 17:52
  • $\begingroup$ I like this question on this stack because it is a basic query about an important concept. I'm glad you've garnered some answers, and welcome to this AI! $\endgroup$ – DukeZhou Aug 29 '17 at 0:47

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.

Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.

Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to do. You just repeat and repeat, until you see the test error no longer improve.


In fact they are telling the same thing: Plot the x-axis as the number of hidden layers, and the y-axis as the performance (e.g. classification accuracy), then this curve will have an upside-down U shape.

Justin's note is clearly saying the same thing as what I wrote above, with added note that the maximum of the curve will happen when x = 2, and Bengio's note is saying the same thing without telling you where the peak could be.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.