Residual Blocks - why do they work?

Question

I've learnt that idea that the residual block was invented to solve the vanishing gradient problem due to the deep layer to layer multiplication.

I understand that for example if I have 10 layers, and I add another 5 layers, that the output of the 10th layer will 'skip' the 5 layers. Although, the output of the 10th layer will also pass through the 5 layers as well. Just before the 15th layer Relu, the output from the 10th layer is element-wise summed with the 15th layer, just prior to the final Relu. I have some confusion with this.

1. Identity mapping/function. I keep reading that it creates an identiy function or it learns an identity function. What exactly is this? Isn't is just F(x) = 5 added layers, and x =output of 10th layer and thus it is just F(X) + X?

2. By summing the output of the 10th layer to the 15th layer, will this not affect what was learnt in the 5 layers? I.e. from 11th -15th layer.

3. I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".

Answer 1

First a little intro, skip to the end for the straight answers: residual networks were proposed after observing that deeper models tend to perform worse than their shallow counterpart if we just keep adding hidden layers without applying any other change to the architecture, as we can see in the very first picture of the original paper.

The reason of this phenomena is indeed gradient vanishing. The more the hidden layers, the more the information of the original input get lost, due to the fact that a hidden layer receives only information from the previous hidden layer. How to solve this? Using residual connections. A residual connection is just an identity function that map an input or hidden state forward in the network, so not to the immediate next layers, that's why these connections are also called skip connections. The only purpose they serve is to force deep layers to retain information learned in the early layers of the network.

From a numerical perspective you can think about information getting lost as weights becoming smaller and smaller. By brutally summing the hidden states of previous layers you make sure to avoid this problem, giving the weights a broader range of adjustment even in very deep layers.

In conclusion, to answer your questions:

1 Residual connection don't create or learn an identity function, they simply use it. The formulation of such connections in the paper is:

$y = F(x, W_{i}) + x$

where x could be rewritten as $I(x)$, $I$ being the identity function.

2 No, we don't loose any information by summing the residuals, on the contrary, they are designed to retaining information also in very deep layers, for the above mentioned reasons.

3 All layers are updated, there's no frozen layers in a residual network. The "skip" term refers to the fact that a hidden layers is copied in forward layers, which is a legit operation, but it doesn't refer at all to skip in training or weight updates.

Answer 2

The residual layer was not invented to solve the vanishing gradient problem. Citing from the official ResNet paper:

An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].

The problem that this layer solves is: How to create deeper models so that they perform definitely better than shallower models?

By adding the residual connection we modify the output of the layer so that instead of $F(x) = f(x)$ it now outputs $F(x) = f(x) + x$, where $f$ is the function represented by the layer around which we add the skip-connection.

(technically the skip-connection is considered the main flow, and the NN layer is called the residual)

Now if you have two identical models, but one has $N$ layers and the other $N+1$ layers, and if the shallower model was the better choice, then the last layer of the deeper model is actually not needed (or one of the layers, no need to be the last). In order for the deeper model to simulate the shallower model it simply has to learn that the function of the last layer has to be $f(x) = 0$, i.e. all of the weights of the last layer have to be $0$. Without the skip-connection the model has to learn that the last layer function has to be $f(x) = x$, which is much more difficult.

So now if your model is not performing well you can simply add more layers and you expect that the performance will not degrade. It may not improve, but it should not degrade.

See this if you want to read more: https://pi-tau.github.io/posts/res-nets/

Answer 3

For the Third question,

I believe it also helps with backpropagation so that it doesn't have to update all the weights layer by layer and it can skip back to shallow layers. Therefore, are the weights inside the residual block, i.e layers 11-15 not updated? If not, then what is the point of the 11-15th layer if they are not designed to "do anything".

Model with many layers, it is easy to get a phenomenon- gradient vanishing. In short, model does not learning in some layers(as layers 11-15 in your example). And Residual Block can help to solve this phenomenon.

So, we can clarify it. These layers are designed and we hope they could work well. But in some situation, not well --> adding Residual Block is a solution.