I was reading about ResNets from this page, and I couldn't understand the following extract, about the motivation behind ResNets:

"Since neural networks are good function approximators, they should be able to easily solve the identity function, where the output of a function becomes the input itself" $$f(x)=x$$ "Following the same logic, if we bypass the input to the first layer of the model to be the output of the last layer of the model, the network should be able to predict whatever function it was learning before with the input added to it." $$f(x) + x = h(x)$$ "The intuition is that learning f(x) = 0 has to be easy for the network."

Learning $f(x)=0$ is probably easy for the network, but what good is it? How does it help?

  • $\begingroup$ the cite from article is basically nonsense, if it was easy for network to learn identity function we wouldn't need residuals, I answered similar question here take a look at it. $\endgroup$ – Brale Apr 19 '20 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.