A class of CNN is popular due to the implementation of residual connections.
We can use both terms "residual connections" and "skip connections" interchangeably as they refer to the same.
Residual connections are the same thing as 'skip connections'. They are used to allow gradients to flow through a network directly, without passing through non-linear activation functions. Non-linear activation functions, by nature of being non-linear, cause the gradients to explode or vanish (depending on the weights).
While studying some models that are used in codes of some deep learning projects, I came across the word "lateral connections".
I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.
Am I true? If not, when can I call a connection lateral?