4
$\begingroup$

I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? E.g., ReLU, ELUs, etc. I'd appreciate any statistics or insight on this.

$\endgroup$
2
$\begingroup$

Currently, both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.

Also, these AFs, more specifically ReLU, are very fast learning AF which makes them even more useful in research.

However, depending on the type of NN you working on, it's always a good practice to pay some attention to new studies.

$\endgroup$
2
  • 1
    $\begingroup$ Maybe you should provide some references that show that ReLU and ELU are the most widely used AFs. If there are some statistics, that would even be better. $\endgroup$
    – nbro
    Jul 27 '20 at 13:24
  • 1
    $\begingroup$ @nbro I will have a look and update any answer I provide over time. I haven't found any good source to provide stat about it. Please let me know if you know any. Thanks for the suggestion and also for keeping this community well. $\endgroup$
    – pedrum
    Jul 28 '20 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.