What is meant by "lateral connection" in the context of neural networks?

Question

A class of CNN is popular due to the implementation of residual connections.

We can use both terms "residual connections" and "skip connections" interchangeably as they refer to the same.

Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).

While studying some models that are used in codes of some deep learning projects, I came across the word "lateral connections".

I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.

Am I true? If not, when can I call a connection lateral?

Answer 1

Lateral connection in NN simply means that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) then lateral connection implies that unit 1 may be connected to unit 2.

So now the activation of unit 2 is not only dependent on incoming inputs but also on unit 1. The weight assigned to a connection between unit 2 and unit 1 needs to be learned as part of backprop.

Please refer to the below image for more details :

Lateral connections in a feed-forward architecture. Inputs and hidden layer neurons are fully connected as are hidden layer and output layer neurons. Neuron j in the hidden layer also receives the net input of neuron (j - 1) in the hidden layer through a lateral connection

Ref: https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0

Lateral Connection can occur between units in layers and can be applied to feed forward network