Why do skip layer connections require the same layer sizes?

Question

I know how skip connections work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow.

My question is, why doesn't it just get implemented in the seemingly more sensible way of concatenating some previous layer's activations onto a later layer's activations?

Most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.

What is the reasoning behind the choice to do this rather than simply using concatenation?

Answer 1

One can concatenate with the previous layer outputs as well, and this approach in pursued in DenseNets. A nice illustration, that compares difference between ResNets and DenseNets is presented below:

As pointed in the other answer it will lead to an increase of computation cost, with the same number of channels (given all other properties of architecture are the same).

Suppose, you had ResNet with the fixed channel size $N$. Standard convolution has computational and storage cost proportional to the product of input and output channels or $O(N^2)$ in the present case.

If you concatenated features from the previous layer, after each layer number of channels would be doubled. Therefore, the computational cost would grow $4$ times for each new layer, and the total cost grows exponentially with depth in this approach.

However, you can make each of the convolution to shrink the number of channels $2$ times and concatenate only half of the channels from the previous layer. In this way, total computational cost and storage cost is the same in every layer.

Answer 2

Of course, it does create more parameters to train, but that seems like a small sacrifice to me.

It is not a "small sacrifice". For the very deep networks that skip connections are applied to, to get the same benefits when concatenating, you would end up witha significant multiplier on the number of parameters.

To get the same passthrough effect on gradient signals (and allow later layers to learn modifications to the identity function), each layer's output would need to be copied to all the following layers. This scales poorly.

Let's take an example of a 10-layer fully-connected network, with 100 neurons per layer in the hidden layers where we want to apply skip connections. In the simple version of this network (ignoring bias to keep the maths simpler), there are 100x100=10,000 parameters for each added layer, making 90,000 parameters overall. If you use addition-based skip connections, the total number of parameters remains the same, at 90,000. If you use concatenation, the layers connect as 100x100, (100+100)x100, (100+100+100)x100 etc, so you end up with 450,000 parameters, five times as many. This is not a "small sacrifice", this is a scaling problem.

The technique of concatenating layers into later parts of the network is known and used. As is adding "early output" or head layers to generate gradient signals at different points in the network. These are valid approaches, and can help with vanishing gradient problems in a similar way. However, the additive skip connections in a residual network scale much better into very deep networks.

Concatenated copies of layers, or additional network heads with the same target functions and loss, are still used, but more sparingly.

This avoids the strange choice of addition, which has a questionable benefit

It may avoid a specific addition mechanism that you seem concerned about. However there are still very similar additions occurring when the concatenated layer feeds forward to the next layer. You could set those weights in a specific way (a copy of the weights of the new layer that it is concatenating with for just two combined layers) when the layer sizes are the same, and the concatenation and addition approaches would very similar.

Answer 3

There is no obvious reason why concatenating layers will "stabilize" the training of the network. In fact, you are adding more information, which might improve accuracy, but perhaps at certain computational disadvantages.

ResNets prevent the vanishing gradient problem by "passing information" from previous layers onto the next. Page 6 of these slides shows calculations that show this to be the case in a specific degenerate edge cases; but as for how it consistently provides such an advantage on more general basis is from what I understand still unclear.