Timeline for If the point of the ResNet skip connection is to let the main path learn the residual relative to identity, why are there convolutional skips?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 20, 2020 at 14:16 | comment | added | user9947 | I think being trainable also doesn't affect much the intuition, since it is basically selecting the projections which is most useful. For example in a car moving problem a projection selecting measurements of acceleration and velocity will be much more useful in predicting the displacement for future states as compared to displacement and velocity (if we know the initial starting position, $s=ut + 0.5at^2$). This is very informal and vague but maybe useful. | |
| Mar 20, 2020 at 14:10 | comment | added | Alexander Soare | I think I'll rewind to before you said anything. I was happy then :P | |
| Mar 20, 2020 at 14:09 | comment | added | user9947 | That's very unlikely. But I was just asking, its a linear connection so being trainable also might not affect much. Can't really say anything. | |
| Mar 20, 2020 at 14:04 | comment | added | Alexander Soare | What if someone got it wrong then everyone copied them... :| | |
| Mar 20, 2020 at 14:03 | comment | added | Alexander Soare |
Here's another implementation in tensorflow. Line 337 defines the projection_shortcut. I'm not too familiar with tf but it looks like a vanilla 1x1 conv to me.
|
|
| Mar 20, 2020 at 14:01 | comment | added | user9947 | It kind of makes a difference, for example in Kalman Filter we use a projection/measurement to rectify a state whose dimensions are not the same. Thus all the information about the correct state is contained in the projection. | |
| Mar 20, 2020 at 13:59 | comment | added | Alexander Soare | If it's not, that will make a big difference to the way I think about this. Maybe I'll go double check by looking at more implementations. They keep calling it a "projection" shortcut in the papers, and to me projection means collapsing one dimension of a vector down to zero, so I'm not sure it relates | |
| Mar 20, 2020 at 13:37 | comment | added | user9947 | Are you sure the $W_s$ is trainable? | |
| Mar 20, 2020 at 12:43 | answer | added | Alexander Soare | timeline score: 3 | |
| Mar 19, 2020 at 17:59 | history | asked | Alexander Soare | CC BY-SA 4.0 |