When are multiple hidden layers necessary?

Question

I know that my question probably seems like being asked many times, but Ill try to be more speciffic:

Limitations to my question:

1. I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)

2. My question is NOT about classification using neural networks

3. I am asking about a "simple" neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.

Preambula:

As far as I understood, from the universal approximation theorem, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here http://neuralnetworksanddeeplearning.com/chap4.html.

Question 1

In this specific case, is there any added value in using extra layers? (maybe the model will be more precise, or faster training?)

Question 2

Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?

Question 3

Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.

Answer 1

A very wide but shallow neural network is going to be harder to train.
You can check that with the playground of tensorflow or with the MPG example in Google Colab.
The relationship between architecture and learning capabilities is not fully understood, but, empirically, thats what you see.
But making the network too deep creates more problems:

• Vanishing gradients.
• Less parameters for the same amount of neurons.
• Exploding gradients.

It is for that reason that humans and neural networks are the one deciding a good architecture.