6
$\begingroup$

The Softsign (a.k.a. ElliotSig) activation function is really simple:

$$ f(x) = \frac{x}{1+|x|} $$

It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally extremely simple (easy for, e.g., a GPU).

Why it is not widely used in neural networks? Is it because it is not infinitely derivable?

$\endgroup$
  • $\begingroup$ GPU is used for parallelizing calculations. I don't see how this would be useful in the choice of activation functions. $\endgroup$ – DuttaA Mar 18 at 15:51
  • 1
    $\begingroup$ @DuttaA - It is because the activation function is part of the calculations. I am just wondering if for a GPU it may be quicker to compute simple algebraic functions instead of exponentials. $\endgroup$ – Pietro Mar 18 at 18:11
  • $\begingroup$ that's a very legitimate question although I wouldn't say GPU will help in this case since we don't parallelize to such fine details like exponential calculation, but you are right calculation of exponentials are quite costly that is why we switched to ReLu $\endgroup$ – DuttaA Mar 18 at 18:13
4
$\begingroup$

I can't speak for individual researchers, but I can guess why the community as a whole hasn't adopted this activation function.

ReLU is just so incredibly cheap. This benefit continues to grow as networks grow deeper. Also, they work reasonably well. As pointed out in Searching for Activation Functions,

the performance improvements of the other activation functions tend to be inconsistent across different models and datasets.

Even if a new activation function did provide a meager improvement in performance across the board, I wouldn't be surprised if ReLU were still commonly used. It's the default for a lot of machine learning software already.

Also research isn't ordinarily about eeking out one more percentage point in accuracy on a specific task. If I were entering in a competition, I might experiment with activation functions. But even then I'd rather use ReLU and save a little time while prototyping my architecture.

As pointed out by @DuttaA in comments, softsign could potentially replace sigmoid and tanh in situations where a bounded output is desired. I haven't seen anyone compare them before, but softsign would at least be much faster. I'm guessing this replacement hasn't happened because of tradition and exposure. Not because of softsign's lack of infinite derivatives. I don't know if this happening would make softsign "widely used", but it would be something.

$\endgroup$
  • $\begingroup$ It matters though what activation you are using depending on the task..For example in NLP esque tasks they exclusively use activations like tanh, sigmoid, etc. Apparently it works well..Also for autoencoders they use exponential functions probably because in the cost function they are using log likelihood. $\endgroup$ – DuttaA Mar 18 at 5:37
  • $\begingroup$ Sure, sigmoid and tanh are used sometimes. But they have specific use cases. And yes, things like the softmax output activation are obviously still used. I didn't mean to imply that they weren't. But most of the time, on a general neuron in the middle of a deep architecture, we choose ReLU. Actually, the proposed activation might best be used as a replacement for sigmoid and tanh, where bounded outputs are important, but I'm speculating. $\endgroup$ – Philip Raeisghasem Mar 18 at 6:17
  • $\begingroup$ That's​the point I'm trying to highlight, they are used extensively in DNNs and CNNs but when it comes to NLPesque tasks like RNNs and autoencoders, the authors of some papers behave like ReLu doesn't exist at all when in reality these structures due to their inherent sequential nature are the most in need to of simple activation. $\endgroup$ – DuttaA Mar 18 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.