# Are there relatively new research papers that describe how to make back-propagation more efficient?

I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any.

Are there relatively new research papers that describe how to make back-propagation more efficient?

So, I am looking for papers similar to Efficient Backprop by LeCun but newer. The papers could describe why ReLU now "dominates" tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.

• I think you meant to say non-homeomorphic mapping rather than discontinuous, as noted sect. 6.2 of the paper by Naitzat et al. you linked. The $\mathrm{ReLU}$ mapping is continuous, of course. – htl Apr 6 at 14:51
• $\mathrm{ReLU}$ isn't differentiable at zero, but it is continuous. Roughly, a continuous function is one that "you can draw without lifting your pen" (the actual definition is more formal of course!). The reason that $\mathrm{ReLU}$ is not a homeomorphism is because it doesn't have a well-defined inverse (lots of values map to 0), and a homeomorphism is required to be invertible, and have a continuous inverse. That's why the authors contrast with the sigmoid, which does have an inverse. – htl Apr 6 at 18:08