For example, hidden layer 1's outputs would be fed to the perceptrons in layer 2, 3, 4, ... etc.

Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?

My intuition is that humans combine simple decisions with more complex ones to form an answer.

Also, wouldn't this solve the vanishing gradient problem?

If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.

  • $\begingroup$ Welcome to ai.se...I find your question a bit unclear..Why not add the question title in the question body itself and specify more specifically want connections you are trying to create instead of the standardized connection... $\endgroup$ – DuttaA Apr 2 '18 at 6:19

Actually it doeas!

I happened to make a presentation of a paper that talks about this topic. They called it DenseNet, which stands for densely connected convolutional networks. Just like in your question, within a denseblock the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed forward neural network the l-th layer is a function of the previous output x_l = H(x_{l-1}), while in the densenet each layer is a function of all the previous outputs x_l = H([x_0, x_1,..., x_{l-1}])


However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between denseblock and pooling layers.

The results are clear: not only in almost all the tests the accuracy of the densenet is greater to that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow to act during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.

If you're interested you should definitely check out their paper.

  • $\begingroup$ There is a similar concept called Resnet, that uses "skip connections" to achieve a similar result. The difference is that instead of direct deep connections, the skip connections are summed into the existings the next but one layer. Mathematically this creates a "short-circuit" for gradient flow to earlier layers when training, and also makes it very easy for later layers to learn the identity function, which intuitively means they can learn improvements over identity function. $\endgroup$ – Neil Slater Apr 2 '18 at 11:07

In regard to your intuition i.e. humans combine simple decisions with more complex ones to form an answer is semantically correct but as in case of FEED-FORWARD neural network , these networks are called feed-forward because the output from one layer of neurons feeds forward into the next layer of neurons. There are never any backward connections, and connections never skip a layer.
Typically, the layers are fully connected, meaning that all units at one layer are connected with all units at the next layer. Connection and relevance exists between the two adjacent layer and that's how a human cognitive mind works. Say for example a baby learns to identify an object by firstly identifying the texture for the object, its shape etc(first layer). Then this result is forwarded to next (layer) for its more-precise detail as color(second layer) and hence finally the output. The process is systematic, sequential and continuous.
Moreover the vanishing gradient problem arise mainly due to the choice of the activation function used. As food for thought its nice to think the way you thought of handling the vanishing gradient problem but since the architecture of Feed-forward is designed in a way to learn things up, push them ahead as raw input for the next and so on.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.