Are there any architectures of deep neural networks that connect input neurons not only with first hidden layer but also with deeper ones (red lines on the picture)?

enter image description here

If so could you give some names or links to research papers?

  • $\begingroup$ This seems to be a duplicate of ai.stackexchange.com/q/17190/2444. Well, at least, I answer your question there too. $\endgroup$ – nbro Oct 29 '20 at 9:17
  • $\begingroup$ @nbro Well, I don't think this question is a duplicate, but yes there is an answer to my question in the linked one. Thanks! $\endgroup$ – Makintosz Oct 29 '20 at 9:41

This type of connections are called skip or residual connections. There are numerous works which employs this type of mechanism, for example: ResNet, SkipRNN. In addition here you can find a paper that empirically explores the skip connections for sequential tagging, or this one for speech enhancement.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.