8
$\begingroup$

The Softsign (a.k.a. ElliotSig) activation function is really simple:

$$ f(x) = \frac{x}{1+|x|} $$

It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally extremely simple (easy for, e.g., a GPU).

Why it is not widely used in neural networks? Is it because it is not infinitely derivable?

Improve this question
$\endgroup$
3
  • $\begingroup$ GPU is used for parallelizing calculations. I don't see how this would be useful in the choice of activation functions. $\endgroup$
    – user9947
    Commented Mar 18, 2019 at 15:51
  • 1
    $\begingroup$ @DuttaA - It is because the activation function is part of the calculations. I am just wondering if for a GPU it may be quicker to compute simple algebraic functions instead of exponentials. $\endgroup$
    – Pietro
    Commented Mar 18, 2019 at 18:11
  • $\begingroup$ that's a very legitimate question although I wouldn't say GPU will help in this case since we don't parallelize to such fine details like exponential calculation, but you are right calculation of exponentials are quite costly that is why we switched to ReLu $\endgroup$
    – user9947
    Commented Mar 18, 2019 at 18:13

1 Answer 1

Reset to default
6
$\begingroup$

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.

ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions,

the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.

Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.

Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.

As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.

Improve this answer
$\endgroup$
3
  • $\begingroup$ It matters though what activation you are using depending on the task..For example in NLP esque tasks they exclusively use activations like tanh, sigmoid, etc. Apparently it works well..Also for autoencoders they use exponential functions probably because in the cost function they are using log likelihood. $\endgroup$
    – user9947
    Commented Mar 18, 2019 at 5:37
  • 2
    $\begingroup$ Sure, sigmoid and tanh are used sometimes. But they have specific use cases. And yes, things like the softmax output activation are obviously still used. I didn't mean to imply that they weren't. But most of the time, on a general neuron in the middle of a deep architecture, we choose ReLU. Actually, the proposed activation might best be used as a replacement for sigmoid and tanh, where bounded outputs are important, but I'm speculating. $\endgroup$
    – Philip Raeisghasem
    Commented Mar 18, 2019 at 6:17
  • $\begingroup$ That's​the point I'm trying to highlight, they are used extensively in DNNs and CNNs but when it comes to NLPesque tasks like RNNs and autoencoders, the authors of some papers behave like ReLu doesn't exist at all when in reality these structures due to their inherent sequential nature are the most in need to of simple activation. $\endgroup$
    – user9947
    Commented Mar 18, 2019 at 6:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .