4
$\begingroup$

In the original ResNet paper they talk about using plain identity skip connections when the input and output of a block have the same dimensions.

enter image description here

When the input and output have different dimensions they propose two options:

(A) Use an identity mapping padded with zeros to make up for the extra dimensions

(B) Use a "projection".

enter image description here

which (after some digging around in other people's code) I see as meaning: do a convolution with a 1x1 kernel with trainable weights.

(B) is confusing to me because it seems to ruin the point of ResNet by making the skip connection trainable. Then the main path is not really learning a "residual" relative to an identity transformation. So at this point, I'm no longer sure how to interpret the intent or expected effect of this type of block. And I would think that one should justify doing it in the first places instead of just not putting a skip connection there at all (which in my mind is the status-quo before this paper).

So can anyone help explain away my confusion here?

Improve this question
$\endgroup$
8
  • $\begingroup$ Are you sure the $W_s$ is trainable? $\endgroup$
    – user9947
    Commented Mar 20, 2020 at 13:37
  • $\begingroup$ If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates $\endgroup$
    – Alexander Soare
    Commented Mar 20, 2020 at 13:59
  • $\begingroup$ It kind of makes a difference, for example in Kalman Filter we use a projection/measurement to rectify a state whose dimensions are not the same. Thus all the information about the correct state is contained in the projection. $\endgroup$
    – user9947
    Commented Mar 20, 2020 at 14:01
  • $\begingroup$ Here's another implementation in tensorflow. Line 337 defines the projection_shortcut. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me. $\endgroup$
    – Alexander Soare
    Commented Mar 20, 2020 at 14:03
  • 1
    $\begingroup$ I think being trainable also doesn't affect much the intuition, since it is basically selecting the projections which is most useful. For example in a car moving problem a projection selecting measurements of acceleration and velocity will be much more useful in predicting the displacement for future states as compared to displacement and velocity (if we know the initial starting position, $s=ut + 0.5at^2$). This is very informal and vague but maybe useful. $\endgroup$
    – user9947
    Commented Mar 20, 2020 at 14:16

1 Answer 1

Reset to default
3
$\begingroup$

Well, I found an answer that satisfies me.

The zero-padded identity is not ideal. Suppose we're mapping from 64 channels to 128 channels. Then the zero-padded identity will map to an output where half of the channels are the same as the inputs, and the other half are all zeros. So that means the main path is learning a residual for half of the output channels, and learning a mapping from the ground up for the other half of the channels.

Now, the alternative is to use option (B) which is the single 1x1 convolution. After reading their ResNetV2 paper I realised that they really double down on this concept of maintaining a "clean" alternative path all the way up and down the network. So then the question is, what's "cleaner"?: a block of two convolutions of the form 3x3x64 then 3x3x128, or a shortcut with a 1x1x128 convolution? The shortcut is a tempting choice here. And in fact, in the original paper they show empirically that the convolutional shortcut is better than the identity shortcut. Maybe it might be worth running a test to see if the convolutional shortcut is better than no shortcut at all, but until I decide to run such a test, I'll presume the authors would have mentioned it. The other thing worth noting is that there are only a few places in the whole network where the dimensions need to be increased, so maybe the impact is small.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .