2
$\begingroup$

Consider a feedforward neural network. Suppose you have a layer of inputs, which is feedforward to a hidden layer, and feedforward both the input and hidden layers to an output layer. Is there a name for this architecture? A layer feeds forward around the layer after it?

$\endgroup$
  • $\begingroup$ How are you combining the two layers - concatenating the two vectors to make one longer vector size num_imputs + num_hidden? $\endgroup$ – Neil Slater Feb 3 at 20:05
2
$\begingroup$

This could be called a residual neural network (ResNet), which is a neural network with skip connections, that is, connections that skip layers.

Here's a screenshot of a figure from the paper Deep Residual Learning for Image Recognition (2015), an important paper that shows the usefulness of these architectures.

enter image description here

| improve this answer | |
$\endgroup$
2
$\begingroup$

Such a network could be either a Residual Network or a Highway Network depending upon the underlying architecture of the skip layers. They are primarily used to to tackle the problem of vanishing gradients in very deep networks by reusing activations from a previous layer and passing them to adjacent layers (two or three skips away).

$z$ acts as a Gating Function, which controls how much information is to be transmitted from a previous layer in the network. An additional weight matrix is used to estimate the skip weights.Highway

This architecture does not involve a Gate Controller. The shortcut connections are realized by adding the outputs of a previous layer to that of the connecting layer.

Residual

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.