Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name for this architecture? A layer feeds forward around the layer after it?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ How are you combining the two layers - concatenating the two vectors to make one longer vector size num_imputs + num_hidden? $\endgroup$– Neil SlaterCommented Feb 3, 2020 at 20:05
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers.
Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.
$\begingroup$
$\endgroup$
Such a network could be either a Residual Network or a Highway Network depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing gradients in very deep networks by reusing activations from a previous layer and passing them to adjacent layers (two or three skips away).
- Highway Network: (Training Very Deep Networks, 2015)
$z$ acts as a Gating Function, which controls how much information is to be transmitted from a previous layer in the network. An additional weight matrix is used to estimate the skip weights.
- Residual Network:(Deep Residual Learning for Image Recognition ,2015)
This architecture does not involve a Gate Controller. The shortcut connections are realized by adding the outputs of a previous layer to that of the connecting layer.
