I know that my question probably seems like being asked many times, but Ill try to be more speciffic:
Limitations to my question:
I am NOT asking about convolutional neural networks, so please, try not to mention this as an example or as an answer as long as it is possible. (maybe only in question number 3)
My question is NOT about classification using neural networks
I am asking about a "simple" neural network designed to solve the regression type of problem. Let's say it has 2 inputs and 1 output.
As far as I understood, from the universal approximation theorem, in such a case, even if the model is nonlinear, only one hidden layer can perfectly fit a nonlinear model, as shown here http://neuralnetworksanddeeplearning.com/chap4.html.
In this specific case, is there any added value in using extra layers? (maybe the model will be more precise, or faster training?)
Suppose in 1st question the answer was there is no added value. In such a case will the added value appear if I enlarge inputs from two inputs as described above, to some larger number?
Suppose in 2nd question the answer was there is no added value. I am still trying to pinpoint the situation where it STARTS making sense in adding more layers AND where it makes NO sense at all using one layer.