100 layer neural network with 100 hidden units vs. 1 layer neural network with 100 hidden units

Question

Suppose we have a neural network with 100 hidden layers. Each hidden layer has one hidden node, and the hidden nodes employ a universal basis function (e.g. tanh). Now we want to compare this network's approximation capacity to another network with one hidden layer with 100 hidden nodes (all using tanh). Can both of these networks approximate the same set of functions?

I have read from a few different sources that a neural network has to have "enough" hidden units, use a universal basis function, and have at least a single hidden layer to approximate any reasonable function to any degree of accuracy. However, I'm not sure what "enough" means. Intuitively, the network with one hidden layer that has 100 hidden units should satisfy this condition but I can't come up with any reasoning to back this up.

Answer 1

The answer to this is so broad as to be unsatisfying. Those with sufficient experience training neural networks know that there isn’t a single architecture that solves a problem and there are some that seemingly never solve a problem.

Further a layer which is wider that the previous transforms the output of the previous to a higher dimension. The alternative is true of smaller layers. If the activation function draws a linear boundary and the data is not linearly separable in its native dimension transforming to a lower dimension (100 layers of 1 neuron) won’t likely be useful. Whereas at a higher dimension it may be linearly separable (1 layer of 100).

Finally, ANN are universal approximators. One can do MNIST classification with dense layers but it becomes a lot easier and more accurate with convolution. Although a dense network can approximate the behavior of CNN there is enough stochasticness in training it isn’t likely to do nearly as good. Neural networks can often do much better if we make assumptions about the problem space. Which may mean a variety of dimensional transformations in a dense network.