What does "linear unit" mean in the names of activation functions?

Question

Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. "Linear unit" can be abbreviated as LU.

For example, consider some activation functions

ELU - Exponential Linear Unit

ReLU - Rectified Linear Unit

................................................

Why does the function name contain the term "Linear Unit"? What is meant by Linear Unit here? Is it saying anything about the nature of function under consideration?

Answer 1

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016):

You can see that these functions are linear functions for $x > 0$, that's why they are called Linear Units.

For example, the ELU is defined as

$$ ELU(x) = \begin{cases} x &\text{if } x > 0\\ \alpha (\exp(x)-1) & \text{if } x \leq 0. \end{cases} $$ These functions introduce the nonlinearity around zero, each in its own way, which can be used for different problems.