1
$\begingroup$

I've learnt that idea that the residual block was invented to solve the vanishing gradient problem due to the deep layer to layer multiplication.

I understand that for example if I have 10 layers, and I add another 5 layers, that the output of the 10th layer will 'skip' the 5 layers. Although, the output of the 10th layer will also pass through the 5 layers as well. Just before the 15th layer Relu, the output from the 10th layer is element-wise summed with the 15th layer, just prior to the final Relu. I have some confusion with this.

  1. Identity mapping/function. I keep reading that it creates an identiy function or it learns an identity function. What exactly is this? Isn't is just F(x) = 5 added layers, and x =output of 10th layer and thus it is just F(X) + X?

  2. By summing the output of the 10th layer to the 15th layer, will this not affect what was learnt in the 5 layers? I.e. from 11th -15th layer.

  3. I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".

$\endgroup$
6
  • $\begingroup$ An identity function is f(x)=x so F(X)+X is the same as F(X)+f(X) if f is the identity function $\endgroup$
    – user253751
    Aug 26, 2021 at 15:17
  • $\begingroup$ @user253751 Yes I know that, but why do we train the residual block to output 0? Because we need it to be 0 such that x the identity, can be the same as the input to the residual block. $\endgroup$
    – user49443
    Aug 27, 2021 at 5:18
  • $\begingroup$ The purpose of a residual block is not to output zero, I am not sure where you got that from. $\endgroup$
    – Dr. Snoopy
    Aug 27, 2021 at 17:20
  • $\begingroup$ @Dr.Snoopy I keep reading that the output should equal to input. And the equation F(x) = f(x) + x keeps getting brought up and they say the point of identity is the input should be the same as the output, so in my head that means the residual block needs to be 0, or learn to make it 0. because F(x) = 0 + x which is F(x) = x $\endgroup$
    – user49443
    Aug 30, 2021 at 9:02
  • $\begingroup$ Keep reading, where? That is not the correct interpretation, I think you are reading about a different kind of residual. $\endgroup$
    – Dr. Snoopy
    Aug 30, 2021 at 11:32

2 Answers 2

2
$\begingroup$

First a little intro, skip to the end for the straight answers: residual networks were proposed after observing that deeper models tend to perform worse than their shallow counterpart if we just keep adding hidden layers without applying any other change to the architecture, as we can see in the very first picture of the original paper. enter image description here

The reason of this phenomena is indeed gradient vanishing. The more the hidden layers, the more the information of the original input get lost, due to the fact that a hidden layer receives only information from the previous hidden layer. How to solve this? Using residual connections. A residual connection is just an identity function that map an input or hidden state forward in the network, so not to the immediate next layers, that's why these connections are also called skip connections. The only purpose they serve is to force deep layers to retain information learned in the early layers of the network.

From a numerical perspective you can think about information getting lost as weights becoming smaller and smaller. By brutally summing the hidden states of previous layers you make sure to avoid this problem, giving the weights a broader range of adjustment even in very deep layers.

In conclusion, to answer your questions:

1 Residual connection don't create or learn an identity function, they simply use it. The formulation of such connections in the paper is:

$y = F(x, W_{i}) + x$

where x could be rewritten as $I(x)$, $I$ being the identity function.

2 No, we don't loose any information by summing the residuals, on the contrary, they are designed to retaining information also in very deep layers, for the above mentioned reasons.

3 All layers are updated, there's no frozen layers in a residual network. The "skip" term refers to the fact that a hidden layers is copied in forward layers, which is a legit operation, but it doesn't refer at all to skip in training or weight updates.

$\endgroup$
2
  • $\begingroup$ Thanks for the detailed description. I have a quick question. I'm reading that the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal. So that is why I am confused that to make this happen you need the residual block to output 0. Also, does summating the input with the output of the residual block not alter the true learning of the residual block? or is the purpose to just mix in information previously learnt with information learnt in the residual block. Thanks. $\endgroup$
    – user49443
    Aug 30, 2021 at 9:00
  • $\begingroup$ "the goal of the residual block so that before the last 'ReLU' activation, the summation of the input and the output of the residual block, should equal ". The summation is not suppose to be zero, the 'skip connection' before the relu actually prevent everything going to zero, due to vanishing gradient. "is the purpose to just mix in information previously learnt with information learnt in the residual block" Yes, precisely. Thanks to the skip connections a deep layer would at the very least output the same as previous layers, but never loose information. $\endgroup$ Aug 31, 2021 at 12:20
0
$\begingroup$

For the Third question,

I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".

Model with many layers, it is easy to get a phenomenon- gradient vanishing. In short, model does not learning in some layers(as layers 11-15 in your example). And Residual Block can help to solve this phenomenon.

So, we can clarify it. These layers are designed and we hope they could work well. But in some situation, not well --> adding Residual Block is a solution.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .