# Why aren't there neural networks that connect the output of each layer to all next layers?

Why aren't there neural networks that connect the output of each layer to all next layers?

For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?

Also, wouldn't this solve the vanishing gradient problem?

If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.

I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $$l$$th layer is a function of the previous output $$x_l = H(x_{l-1})$$, while in the dense net each layer is a function of all the previous outputs $$x_l = H([x_0, x_1, \dots, x_{l-1}])$$.