New answers tagged activation-function
1
Your interpretation is definitely correct. As you correctly pointed out, the derivative of softplus is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for ReLU.
What is quite interesting here is why softplus can be called an approximation to ReLU.
If we break down the definition of softplus, we note that the ...
6
Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016):
You can see that these functions are linear functions for $x > 0$, that's why they are called Linear Units.
For example, the ELU is defined as
$$ ELU(x) = \begin{cases}
x &\text{if } x > 0\\
\alpha (\exp(x)-1) & \text{if } x \leq 0.
\end{cases} $...
3
A smooth function is usually defined to be a function that is $n$-times continuously differentiable, which means that $f$, $f'$, $\dots$, $f^{(n - 1)}$ are all differentiable and $f^{(n)}$ is continuous. Such functions are also called $C^n$ functions.
It can be a bit of a vague term; some people might even stretch the definition and say any continuous ...
Top 50 recent answers are included
