9
$\begingroup$

Why aren't there neural networks that connect the output of each layer to all next layers?

For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?

Also, wouldn't this solve the vanishing gradient problem?

If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
5
$\begingroup$

Actually, this already exists!

I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $l$th layer is a function of the previous output $x_l = H(x_{l-1})$, while in the dense net each layer is a function of all the previous outputs $x_l = H([x_0, x_1, \dots, x_{l-1}])$.

(imagine)

However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.

The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.

If you're interested you should definitely check out their paper Densely Connected Convolutional Networks (2018).

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ There is a similar concept called Resnet, that uses "skip connections" to achieve a similar result. The difference is that instead of direct deep connections, the skip connections are summed into the existings the next but one layer. Mathematically this creates a "short-circuit" for gradient flow to earlier layers when training, and also makes it very easy for later layers to learn the identity function, which intuitively means they can learn improvements over identity function. $\endgroup$
    – Neil Slater
    Apr 2, 2018 at 11:07
  • 1
    $\begingroup$ @NeilSlater Although this is an old question and residual networks are older than dense nets, I think it's worth adding your answer that talks about residual networks. $\endgroup$
    – nbro
    Mar 11, 2020 at 0:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .