1
$\begingroup$

I know how skip connections work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow.

My question is, why doesn't it just get implemented in the seemingly more sensible way of concatenating some previous layer's activations onto a later layer's activations?

Most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.

What is the reasoning behind the choice to do this rather than simply using concatenation?

$\endgroup$
6
  • 1
    $\begingroup$ It's a good question, but why does addition have a "questionable benefit"? If you can explain why you think that, it may help with an answer. As it stands, it is not useful in the question, because it seems to just be an opinion, so cannot be answered. $\endgroup$ Aug 13 at 20:19
  • $\begingroup$ Because most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful. $\endgroup$
    – profPlum
    Aug 13 at 20:35
  • $\begingroup$ Thanks for the response. It would be helpful to edit that into the question. $\endgroup$ Aug 13 at 20:37
  • $\begingroup$ Is your question why adding the output of layers vs concatenating or why the same shape? Either way, the spatial dimensions (in CNN) must have the same shape. As far as I know, skip connections utilize concatenation, whereas ResNet addition (long vs short skip connections) $\endgroup$ Aug 14 at 0:35
  • $\begingroup$ It seems to me that there are 2 distinct questions in this post (the one in the title and the other in the body of the post). Please, edit your post to leave just one question. $\endgroup$
    – nbro
    Aug 14 at 10:49
2
$\begingroup$

One can concatenate with the previous layer outputs as well, and this approach in pursued in DenseNets. A nice illustration, that compares difference between ResNets and DenseNets is presented below: enter image description here

As pointed in the other answer it will lead to an increase of computation cost, with the same number of channels (given all other properties of architecture are the same).

Suppose, you had ResNet with the fixed channel size $N$. Standard convolution has computational and storage cost proportional to the product of input and output channels or $O(N^2)$ in the present case.

If you concatenated features from the previous layer, after each layer number of channels would be doubled. Therefore, the computational cost would grow $4$ times for each new layer, and the total cost grows exponentially with depth in this approach.

However, you can make each of the convolution to shrink the number of channels $2$ times and concatenate only half of the channels from the previous layer. In this way, total computational cost and storage cost is the same in every layer.

$\endgroup$
3
  • $\begingroup$ I don't see how it would grow exponentially... You're correct that it could grow if you used the output the concat operation directly in the next skip layer connection. But the idea is that you would instead use the output of the next layer as skip connection input. That being said I see that this growing structure is what Dense net is trying to implement, that is interesting... $\endgroup$
    – profPlum
    Aug 16 at 21:26
  • $\begingroup$ But even if you allowed it to grow the number of parameters would grow linearly, and then plugging into O(N^2) would give you quadratic growth right? $\endgroup$
    – profPlum
    Aug 16 at 21:31
  • $\begingroup$ @profPlum it depends on how you craft the layers. Typically, one would double the number of feature maps each next layer, which leads to exponential growth. $\endgroup$ Aug 17 at 1:07
0
$\begingroup$

Of course, it does create more parameters to train, but that seems like a small sacrifice to me.

It is not a "small sacrifice". For the very deep networks that skip connections are applied to, to get the same benefits when concatenating, you would end up witha significant multiplier on the number of parameters.

To get the same passthrough effect on gradient signals (and allow later layers to learn modifications to the identity function), each layer's output would need to be copied to all the following layers. This scales poorly.

Let's take an example of a 10-layer fully-connected network, with 100 neurons per layer in the hidden layers where we want to apply skip connections. In the simple version of this network (ignoring bias to keep the maths simpler), there are 100x100=10,000 parameters for each added layer, making 90,000 parameters overall. If you use addition-based skip connections, the total number of parameters remains the same, at 90,000. If you use concatenation, the layers connect as 100x100, (100+100)x100, (100+100+100)x100 etc, so you end up with 450,000 parameters, five times as many. This is not a "small sacrifice", this is a scaling problem.

The technique of concatenating layers into later parts of the network is known and used. As is adding "early output" or head layers to generate gradient signals at different points in the network. These are valid approaches, and can help with vanishing gradient problems in a similar way. However, the additive skip connections in a residual network scale much better into very deep networks.

Concatenated copies of layers, or additional network heads with the same target functions and loss, are still used, but more sparingly.

This avoids the strange choice of addition, which has a questionable benefit

It may avoid a specific addition mechanism that you seem concerned about. However there are still very similar additions occurring when the concatenated layer feeds forward to the next layer. You could set those weights in a specific way (a copy of the weights of the new layer that it is concatenating with for just two combined layers) when the layer sizes are the same, and the concatenation and addition approaches would very similar.

$\endgroup$
0
$\begingroup$

There is no obvious reason why concatenating layers will "stabilize" the training of the network. In fact, you are adding more information, which might improve accuracy, but perhaps at certain computational disadvantages.

ResNets prevent the vanishing gradient problem by "passing information" from previous layers onto the next. Page 6 of these slides shows calculations that show this to be the case in a specific degenerate edge cases; but as for how it consistently provides such an advantage on more general basis is from what I understand still unclear.

$\endgroup$
1
  • $\begingroup$ tl;dr: read page 6 of those slides $\endgroup$ Sep 10 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.