I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem.
The original 2015 paper contains the following passage in section 4.1:
We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish. In fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works to some extent.
So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.
I'm starting with the initial knowledge that it's "easier" to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it "easier"? And why does the "plain network" do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is "residual". But it just isn't able to.