# If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?

In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions.

When the input and output have different dimensions they propose two options:

(A) Use an identity mapping padded with zeros to make up for the extra dimensions

(B) Use a "projection".

which (after some digging around in other people's code) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.

(B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).

So can anyone help explain away my confusion here?

• Are you sure the $W_s$ is trainable?
– user9947
Mar 20 '20 at 13:37
• If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates Mar 20 '20 at 13:59
• It kind of makes a difference, for example in Kalman Filter we use a projection/measurement to rectify a state whose dimensions are not the same. Thus all the information about the correct state is contained in the projection.
– user9947
Mar 20 '20 at 14:01
• Here's another implementation in tensorflow. Line 337 defines the projection_shortcut. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me. Mar 20 '20 at 14:03
• I think being trainable also doesn't affect much the intuition, since it is basically selecting the projections which is most useful. For example in a car moving problem a projection selecting measurements of acceleration and velocity will be much more useful in predicting the displacement for future states as compared to displacement and velocity (if we know the initial starting position, $s=ut + 0.5at^2$). This is very informal and vague but maybe useful.
– user9947
Mar 20 '20 at 14:16