An exponential linear unit (as proposed by Clevert et al.) uses the function:
(Sorry--would have used MathJax were it available.)
ELUα(x) = α(ex - 1) (if x<0), or x (if x≥0)
Now, this is continuous at x=0, which is great. It's differentiable there too if α=1, which is the value that the paper used to test ELU units.
But if α≠1 (as in the above diagram), then it's no longer differentiable at x=0. It has a crook in it, which seems weird to me. Having your function be differentiable at all points seems advantageous. Further, it seems that if you just make the linear portion evaluate to αx rather than x, that it would be differentiable there. Is there a reason that the function wasn't defined to do this? Or did they not bother, because α=1 is definitely the hyperparameter to use?