3
$\begingroup$

I am not asking what activation function is better. I want to know what activation functions are more used in research or deployment. Also, are they used in combination? e.g. ReLU, ELUs, etc. I'd appreciate any statistic on this or insight.

$\endgroup$
2
$\begingroup$

Currently both ReLU and ELUs are the most popular activation functions (AF) used in neural nets (NNs). This is because they eliminate the vanishing gradient problem that causes major problems in the training process and degrades the accuracy and performance of NN models.

Also these AFs, more specifically ReLU, are very fast learning AF which make them even more useful in research.

However, depending on type of NN you working on, it's always a good practice to pay some attentions to new studies.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Maybe you should provide some references that show that ReLU and ELU are the most widely used AFs. If there are some statistics, that would even be better. $\endgroup$ – nbro Jul 27 at 13:24
  • 1
    $\begingroup$ @nbro I will have a look and update any answer I provide over time. I haven't found any good source to provide stat about it. Please let me know if you know any. Thanks for the suggestion and also for keeping this community well. $\endgroup$ – pedrum Jul 28 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.