2
$\begingroup$

I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how come ReLU is very popular in most NNs (which is even more odd when the authors didn't mention about ReLU at all)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Are you reading the original 1998 paper? That would be my guess as to why ReLU isn't mentioned. $\endgroup$
    – David Hoelzer
    Commented Jul 5, 2022 at 0:02
  • 2
    $\begingroup$ Does this answer your question? Choices of activation functions $\endgroup$
    – Martino
    Commented Jul 6, 2022 at 8:43

1 Answer 1

Reset to default
2
$\begingroup$

For a discussion about the advantages of ReLU, see the original paper by Glorot (2011) "Deep sparse rectifier neural networks". "Efficient Backprop" is a 1998 paper. At the time the use of rectifiers was uncommon and sigmoid was the standard choice of activation.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .