0
$\begingroup$

Consider the following details regarding Softplus activation function

$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$

SoftPlus is a smooth approximation to the ReLU function and can be used to constrain the output of a machine to always be positive.

It says that Softplus is a smooth approximation to the ReLU function. Le us consider the analytical form and plot of RELU function.

$$\text{ReLU}(x)=(x)^+=\max(0,x)$$

enter image description here

The plot of Softplus function is

enter image description here

If we observe the both plots, we can see the Softplus is almost similar to ReLU. There is a property for Softplus that ReLU does not have. ReLU is not differentiable at zero and the derivative of ReLU is also not continuous.

If we observe the behavior of Softplus, it is $n-$times continuously differentiable and hence a smooth function.

Since Softplus is both a smooth function and approximates ReLU, it is considered as a smooth approximation of ReLU.

Is my interpretation correct? if no, then what is meant by "smooth approximation" here?

$\endgroup$
1
$\begingroup$

Your interpretation is definitely correct. As you correctly pointed out, the derivative of softplus is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for ReLU.

What is quite interesting here is why softplus can be called an approximation to ReLU.

If we break down the definition of softplus, we note that the parameter $\beta$ controls the smoothness of the function. In fact, we can define a temperature $T = \frac{1}{\beta}$ (much like the softmax function). Let's denote with $f_T$ a softplus function with temperature $T$. We can easily see that as $T \rightarrow 0$ (or equivalently $\beta \rightarrow \infty$), $f_T(x) \rightarrow \max(0, x)$. In fact, let's compute the limit of the softplus for all $x > 0$:

$\lim_{\beta \rightarrow \infty} \frac{1}{\beta} ln(1 + e^{\beta x})$

Rewrite $ln(1+e^{\beta x})$ as $ln(e^{-\beta x} + 1) +ln(e^{\beta x})$ we get:

$\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(e^{-\beta x} + 1) +ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(0 + 1) + ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[0 + ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}\beta x = \\ = x$

Computing the limit for $x \leq 0 $ is straightforward and the result is $0$. So we see that for high value of $\beta$ (or low value of $T$), the softplus is actually the $\max(0, x)$ function!

Therefore, softplus is a family of functions $f_T$ which approximate ReLU with high precision as $T$ gets to 0 (less smoothness), and low precision with high $T$ (more smoothness).

Among all, we usually use softplus with $beta = 1$ (i.e, $f_1$), and this is why we call it a smooth approximation of the ReLU. Because $f_1(x)$ is smooth ($T=1$) and approximates $f_\infty(x) = \max(0, x) = \text{ReLU}(x)$.

Hope this was useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.