1
$\begingroup$

I know that my question probably seems like being asked many times, but Ill try to be more speciffic:

Limitations to my question:

  1. I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)

  2. My question is NOT about classification using neural networks

  3. I am asking about a "simple" neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.

Preambula:

As far as I understood, from the universal approximation theorem, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here http://neuralnetworksanddeeplearning.com/chap4.html.

Question 1

In this specific case, is there any added value in using extra layers? (maybe the model will be more precise, or faster training?)

Question 2

Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?

Question 3

Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.

$\endgroup$
1
  • 1
    $\begingroup$ Please, ask one question per post. If you have multiple questions, ask each of them in its separate post. Take a look at ai.stackexchange.com/help/on-topic for more info about our site. $\endgroup$
    – nbro
    Oct 9, 2020 at 8:36

1 Answer 1

1
$\begingroup$

A very wide but shallow neural network is going to be harder to train.
You can check that with the playground of tensorflow or with the MPG example in Google Colab.
The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.
But making the network too deep creates more problems:

  • Vanishing gradients.
  • Less parameters for the same amount of neurons.
  • Exploding gradients.

It is for that reason that humans and neural networks are the one deciding a good architecture.

$\endgroup$
1
  • 2
    $\begingroup$ "understud" => understood. Also, vanishing gradients typically occur as you increase the depth and not width? $\endgroup$
    – nbro
    Oct 9, 2020 at 14:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .