Is my understanding on "smooth approximation" correct?

Question

Consider the following details regarding Softplus activation function

$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$

SoftPlus is a smooth approximation to the ReLU function and can be used to constrain the output of a machine to always be positive.

It says that Softplus is a smooth approximation to the ReLU function. Let us consider the analytical form and plot of the RELU function.

$$\text{ReLU}(x)=(x)^+=\max(0,x)$$

The plot of Softplus function is

If we observe both plots, we can see the Softplus is almost similar to ReLU. There is a property for Softplus that ReLU does not have. ReLU is not differentiable at zero and the derivative of ReLU is also not continuous.

If we observe the behavior of Softplus, it is $n-$times continuously differentiable and hence a smooth function.

Since Softplus is both a smooth function and approximates ReLU, it is considered as a smooth approximation of ReLU.

Is my interpretation correct? if no, then what is meant by "smooth approximation" here?

Answer 1

Your interpretation is definitely correct. As you correctly pointed out, the derivative of softplus is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for ReLU.

What is quite interesting here is why softplus can be called an approximation to ReLU.

If we break down the definition of softplus, we note that the parameter $\beta$ controls the smoothness of the function. In fact, we can define a temperature $T = \frac{1}{\beta}$ (much like the softmax function). Let's denote with $f_T$ a softplus function with temperature $T$. We can easily see that as $T \rightarrow 0$ (or equivalently $\beta \rightarrow \infty$), $f_T(x) \rightarrow \max(0, x)$. In fact, let's compute the limit of the softplus for all $x > 0$:

$\lim_{\beta \rightarrow \infty} \frac{1}{\beta} ln(1 + e^{\beta x})$

Rewrite $ln(1+e^{\beta x})$ as $ln(e^{-\beta x} + 1) +ln(e^{\beta x})$ we get:

$\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(e^{-\beta x} + 1) +ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(0 + 1) + ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}[0 + ln(e^{\beta x})] = \\ = \lim_{\beta \rightarrow \infty} \frac{1}{\beta}\beta x = \\ = x$

Computing the limit for $x \leq 0 $ is straightforward and the result is $0$. So we see that for high value of $\beta$ (or low value of $T$), the softplus is actually the $\max(0, x)$ function!

Therefore, softplus is a family of functions $f_T$ which approximate ReLU with high precision as $T$ gets to 0 (less smoothness), and low precision with high $T$ (more smoothness).

Among all, we usually use softplus with $beta = 1$ (i.e, $f_1$), and this is why we call it a smooth approximation of the ReLU. Because $f_1(x)$ is smooth ($T=1$) and approximates $f_\infty(x) = \max(0, x) = \text{ReLU}(x)$.

Hope this was useful.