For example, hidden layer 1's outputs would be fed to the perceptrons in layer 2, 3, 4, ... etc.
Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?
My intuition is that humans combine simple decisions with more complex ones to form an answer.
Also, wouldn't this solve the vanishing gradient problem?
If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.
Actually it doeas!
I happened to make a presentation of a paper that talks about this topic. They called it DenseNet, which stands for densely connected convolutional networks. Just like in your question, within a denseblock the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed forward neural network the l-th layer is a function of the previous output x_l = H(x_{l-1}), while in the densenet each layer is a function of all the previous outputs x_l = H([x_0, x_1,..., x_{l-1}])
However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between denseblock and pooling layers.
The results are clear: not only in almost all the tests the accuracy of the densenet is greater to that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow to act during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.
If you're interested you should definitely check out their paper.
In regard to your intuition i.e. humans combine simple decisions with more complex ones to form an answer is semantically correct but as in case of FEED-FORWARD neural network , these networks are called feed-forward because the output from one layer of neurons feeds forward
into the next layer of neurons. There are never any backward connections, and connections never skip a layer.
Typically, the
layers are fully connected, meaning that all units at one layer are connected with all units at the next layer. Connection and relevance exists between the two adjacent layer and that's how a human cognitive mind works. Say for example a baby learns to identify an object by firstly identifying the texture for the object, its shape etc(first layer). Then this result is forwarded to next (layer) for its more-precise detail as color(second layer) and hence finally the output. The process is systematic, sequential and continuous.
Moreover the vanishing gradient problem arise mainly due to the choice of the activation function used. As food for thought its nice to think the way you thought of handling the vanishing gradient problem but since the architecture of Feed-forward is designed in a way to learn things up, push them ahead as raw input for the next and so on.
