Timeline for Residual Blocks - why do they work?
Current License: CC BY-SA 4.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 31, 2021 at 12:20 | comment | added | Edoardo Guerriero | "the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal ". The summation is not suppose to be zero, the 'skip connection' before the relu actually prevent everything going to zero, due to vanishing gradient. "is the purpose to just mix in information previously learnt with information learnt in the residual block" Yes, precisely. Thanks to the skip connections a deep layer would at the very least output the same as previous layers, but never loose information. | |
| Aug 30, 2021 at 9:00 | comment | added | user49443 | Thanks for the detailed description. I have a quick question. I'm reading that the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal. So that is why I am confused that to make this happen you need the residual block to output 0. Also, does summating the input with the output of the residual block not alter the true learning of the residual block? or is the purpose to just mix in information previously learnt with information learnt in the residual block. Thanks. | |
| Aug 27, 2021 at 20:15 | history | edited | Edoardo Guerriero | CC BY-SA 4.0 |
edited body
|
| Aug 27, 2021 at 13:32 | history | answered | Edoardo Guerriero | CC BY-SA 4.0 |