Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s). More neurons creates a more complex function (and thus the ability to model more nuanced decision barriers) than a hidden layer with less nodes.
But what of the hidden layers? What do more hidden layers correspond to in terms of the model generated?
This is a very interesting question that you ask. I believe, this post and this post (written by me) well address your question. However, it deserves an explanation here.
1. Fully connected networks
The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear separation", the first known architecture to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early ResNet in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. Here, however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.
2. Convolutional nets (CNNs)
An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as convolutional layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers "see" more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.
3. Recurrent nets (RNNs)
RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.
To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as ResNets and backpropagation through time, it is possible to train neural networks with a large number of hidden layers.
More hidden layers will just escalate the possibilities amount the neurons, including the solutions from the previous hidden layers. (I will edit this once I am at home and provide you with a good link I found some time ago)
Meanwhile maybe this will help you https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute
It was proven a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions (see Universal approximation theorem).
More layers can't improve something that can already do "everything". But adding more layers reduces the number of necessary neurons, and reduces computing power needed for the network as well.
