2
$\begingroup$

What rectifier is better in general case of Convolutional Neural Network and how about empirical rules to use each type?

  • ReLU
  • PReLU
  • RReLU
  • ELU
  • Leacky ReLU
$\endgroup$
4
$\begingroup$

I've read all the papers about PReLU, LeakyReLU (...) and all the claims how it improves this and that but the little dirty secret is: most of the time it doesn't matter at all and you can't go much wrong with ReLU - empirically proven. I've personally tried all of them in many different problems (from training small networks from scratch through changing activations in large pretrained models) My guess is that gradient doesn't die much in any of them and the rest is pretty much irrelevant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.