2
$\begingroup$

Normaly, the output layer is only connected to the second last layer. Is there any model that the output layer is connected to multiple layers (For example, the second last layer AND the layer before it.)

$\endgroup$
2
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Jun 25, 2022 at 17:35
  • 1
    $\begingroup$ Residual networks have connections that skip layers, the obviously called skip connections. I don't know if skip connections also happen for that specific case you're mentioning, but you should look into ResNets because they may be one answer to your question. $\endgroup$
    – nbro
    Jun 26, 2022 at 9:20

1 Answer 1

1
$\begingroup$

Normaly, the output layer is only connected to the second last layer.

This is only normal for standard MLPs. As soon as you look into slightly more sophisticated neural networks, you'll find a variety of different schemes to connect the layers within a neural network.

Is there any model that the output layer is connected to multiple layers

There are many examples of such a case, like WaveNet, U-Net, HRNet, to name a just few. The first two networks use skip-connections. Mostly summation or concatenation is applied to combine the multilayer results. And like @nbro mentioned in his comment, ResNet is also comparable which uses residual connections to bypass layers.

There are a lot more examples than that because this practice has a very beneficial aspect to it: It shortcuts the backward pass and thus mitigates the vanishing gradient problem.

A more general answer to your question is that you can wire up your layers however you want as long as the network stays differentiable. It should somehow make sense though: In the case of Wavenet these skip-connections help to capture temporal features from different granularities (short-term temporal features detected in the early layers / long-term temporal features detected in later layers).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .