I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any.

Are there relatively new research papers that describe how to make back-propagation more efficient?

So, I am looking for papers similar to Efficient Backprop by LeCun but newer. The papers could describe why ReLU now "dominates" tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.


Here is a paper that explains why ReLU rules.

What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it here.

  • $\begingroup$ I think you meant to say non-homeomorphic mapping rather than discontinuous, as noted sect. 6.2 of the paper by Naitzat et al. you linked. The $\mathrm{ReLU}$ mapping is continuous, of course. $\endgroup$ – htl Apr 6 at 14:51
  • $\begingroup$ Is it? What happens at 0? If I can't map A to B, then, that creates a non-homeomorphic mapping, is that also not discontinuous? Isn't non-homeomorphism discontinuity in terms of topological spaces. $\endgroup$ – Abhishek Verma Apr 6 at 17:20
  • $\begingroup$ $\mathrm{ReLU}$ isn't differentiable at zero, but it is continuous. Roughly, a continuous function is one that "you can draw without lifting your pen" (the actual definition is more formal of course!). The reason that $\mathrm{ReLU}$ is not a homeomorphism is because it doesn't have a well-defined inverse (lots of values map to 0), and a homeomorphism is required to be invertible, and have a continuous inverse. That's why the authors contrast with the sigmoid, which does have an inverse. $\endgroup$ – htl Apr 6 at 18:08
  • $\begingroup$ But, what it means physically is that sigmoid will always remain a lake and you would not be able to cut water. On the other hand, ReLU will be able to create those boundaries since it nullifies everything. Since, topology is the study of shapes. How would you explain this to a novice, the difference between homeomorphism and non-homeomorphism? $\endgroup$ – Abhishek Verma Apr 6 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.