A class of CNN is popular due to the implementation of residual connections.

We can use both terms "residual connections" and "skip connections" interchangeably as they refer to the same.

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).

While studying some models that are used in codes of some deep learning projects, I came across the word "lateral connections".

I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.

Am I true? If not, when can I call a connection lateral?

  • $\begingroup$ When you say "While studying some models that are used in codes of some deep learning projects", it may be a good idea to provide the source where you read about this term. $\endgroup$
    – nbro
    Jan 6 at 13:00

Lateral connection in NN simply mean that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) than lateral connection implies that unit 1 may be connected to unit 2.

So now the activation of unit 2 is not only dependent on incoming inputs but also unit 1. The weight assigned to connection between unit 2 and unit 1 needs to be learned as part of backprop.

Please refer to below image for more details :

enter image description here

Lateral connections in a feed-forward architechture. Inputs, and hidden layer neurons are fully connected as are hidden layer and output layer neurons. Neuron j in the hidden layer also receives the net input of neuron (j - 1) in the hidden layer through a lateral connection

Ref : https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0

Lateral Connection can occur between units in layers and can be applied to feed forward network


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.