Why don't ELUs multiply the linear portion by $\alpha$?

Question

An exponential linear unit (as proposed by Clevert et al.) uses the function:

\begin{align} \text{ELU}_\alpha(x) = \begin{cases} \alpha(e^x - 1), &\text{if } x < 0\\ x, \text{if} &\text{if } x \geq 0\\ \end{cases} \end{align}

Here's a picture.

Now, this is continuous at $x=0$, which is great. It's differentiable there too if $\alpha=1$, which is the value that the paper used to test ELU units.

But if $\alpha \neq $ (as in the above diagram), then it's no longer differentiable at $x=0$. It has a crook in it, which seems weird to me. Having your function be differentiable at all points seems advantageous. Further, it seems that, if you just make the linear portion evaluate to $\alpha x$ rather than $x$, it would be differentiable there.

Is there a reason that the function wasn't defined to do this? Or did they not bother, because $\alpha = 1$ is definitely the hyperparameter to use?

Answer 1

Probably not worth the hassle.

I guess that simply nobody cares. I'm all for having my function be differentiable at all points, but I don't think that having a single jump in the derivative matters much. The neuron basically either repeats its input (x>0) or outputs nearly nothing (x<0) and the point in between is not that important.

Replacing

ELU(x) = x > 0 ? x : α * (exp(x) - 1)

by your proposal

ELU(x) = x > 0 ? α * x : α * (exp(x) - 1)

would have far more consequences. For example, after n layers, the output would get multiplied by α**n, assuming all x on the path being positive. This could be countered by dividing each weight by α, but the derivatives would change. You surely could adjust everything (initial rates, learn rate and whatever), so it'd work as usual, but it could laborious, especially when testing multiple learning algorithms.

I'd suggest to use

ELU(x) = x > 0 ? x : α * (exp(x/α) - 1)

instead, as it just changes the less important part of the curve.

Or did they not bother, because α=1 is definitely the hyperparameter to use?

I can imagine that everyone tried α=1 and was happy with it. It prevented the gradient from being zero and that's it.

I don't have any support of my claim (having never read about something doesn't mean that it doesn't get written about).