6
$\begingroup$

I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem.

The original 2015 paper contains the following passage in section 4.1:

We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with BN, which ensures forward propagated signals to have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN. So neither forward nor backward signals vanish. In fact, the 34-layer plain net is still able to achieve competitive accuracy, suggesting that the solver works to some extent.

So what's happened since then? I feel like either it became a misconception that ResNets solve the vanishing gradient problem (because it does indeed feel like a sensible explanation that one would readily accept and continue to propagate), or some paper has since proven that this is indeed the case.

I'm starting with the initial knowledge that it's "easier" to learn the residual mapping for a convolutional block than it is to learn the whole mapping. So my question is on the level of: why is it "easier"? And why does the "plain network" do such a good job but then struggle to close the gap to the performance of ResNet. Supposedly if the plain network has already learned reasonably good mappings, then all it has left to learn to close the gap is "residual". But it just isn't able to.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get hidden representation $H(x) = F(x) + x$ than it is to learn hidden representation $H(x)$ directly from original representation. That's the main reason and empirical results show that they might be right. Better gradient propagation might be an additional bonus but that's not why they originally introduced the idea.

"Normal" networks work too but, at some point, they become too deep and start working worse than shallower versions (they empirically showed that in the paper). Again, they argue that the reason for that might be that at deeper layers hidden representations become approximately similar $H_n \approx H_{n+1}$ because representation is already well learned and you only need some slight adjustments. That would mean that transformation for deeper layers is similar to identity transformation and that ordinary layers might have trouble learning that, while for residual blocks it would be easy to learn slight modification and add that to the already existing representation from the previous layer.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! I'm almost there. It's just that I wonder why the non-residual version gets close to good mapping but doesn't quite get it. I feel like maybe it's because it settles on a minimum in hp space which is not the correct one, and actually quite far from the correct one. Whereas maybe when mapping a residual, you get dropped in at least the same town as the global minimum, so any local minimum you find is just a few blocks away. And this explanation doesn't rely on vanishing gradients at all. $\endgroup$
    – Alexander Soare
    Commented Mar 20, 2020 at 14:19
  • 3
    $\begingroup$ Well I don't know that's a lot of speculation. You shouldn't take all of this to literally its just an interpretation. The way I looked at it, if you look at the identity operator, that is an identity matrix with ones on diagonal and 0 elsewhere. Now if that matrix represents weights, then diagonal weight should be really large (around 1) and others should be very small (around 0). Those are not very balanced weight relations so NN might have trouble learning that. With residuals you wouldn't have that problem because you're adding to original representation so all weights would be small. $\endgroup$
    – Brale
    Commented Mar 20, 2020 at 17:45
  • 1
    $\begingroup$ I like the idea of identity propagator! In quantum mechanics the time evolution operator for very short time can be expressed as the sum of an identity operator and the Hamiltonian. So in the very short time intervals after the time evolution starts, the system's evolution can be expressed by an identity operator and Hamiltonian operator. In neural networks, I can imagine that as we go from layer to another layer, the representations or weights of layers change gradually which hints the next representation can be obtained by an identity operator and some nonlinear operator. $\endgroup$
    – MOON
    Commented Mar 6, 2022 at 23:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .