3
$\begingroup$

Activation functions, in neural networks, are used to introduce non-linearity. Many activation functions that are used in neural networks have the term "Linear Unit" in their full form. "Linear unit" can be abbreviated as LU.

For example, consider some activation functions

ELU - Exponential Linear Unit

ReLU - Rectified Linear Unit

................................................

Why does the function name contain the term "Linear Unit"? What is meant by Linear Unit here? Is it saying anything about the nature of function under consideration?

$\endgroup$
7
$\begingroup$

Have a look at these graphics showing popular linear units (image taken from Clevert et al. 2016): enter image description here

You can see that these functions are linear functions for $x > 0$, that's why they are called Linear Units.

For example, the ELU is defined as

$$ ELU(x) = \begin{cases} x &\text{if } x > 0\\ \alpha (\exp(x)-1) & \text{if } x \leq 0. \end{cases} $$ These functions introduce the nonlinearity around zero, each in its own way, which can be used for different problems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.