3
$\begingroup$

In this note Justin Domke says that

In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.

But in Bengio's remark, he says

Very simple. Just keep adding layers until the test error does not improve anymore.

There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?

$\endgroup$
2
$\begingroup$

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.

Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.

Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to do. You just repeat and repeat, until you see the test error no longer improve.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Your first link is from 2011, which essentially predates the current deep learning explosion. In the many years that have since passed (AlexNet 2012, ResNet 2015) we have since found that if you keep adding layers, we generally do see improved performance.

enter image description here

This is due to improved training techniques and optimization breakthroughs (residual connections, ReLU, dropout etc.). But do note that the result can be diminishing. In particular, take a look at Deep Equillibrium Models, which essentially allow us to train (in the limit equivalence) infinite depth neural networks.

| improve this answer | |
$\endgroup$
-1
$\begingroup$

In fact they are telling the same thing: Plot the x-axis as the number of hidden layers, and the y-axis as the performance (e.g. classification accuracy), then this curve will have an upside-down U shape.

Justin's note is clearly saying the same thing as what I wrote above, with added note that the maximum of the curve will happen when x = 2, and Bengio's note is saying the same thing without telling you where the peak could be.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.