2
$\begingroup$

According to this blog post

The purpose of an activation function is to add some kind of non-linear property to the function

The sigmoid is typically used as an activation function of a unit of a neural network in order to introduce non-linearity.

Is ReLU a non-linear activation function? And why? If not, then why is it used as an activation function of neural networks?

$\endgroup$
2
$\begingroup$

Short Answer: Yes

Visually:

enter image description here

if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia:

"In calculus and related areas, a linear function is a function whose graph is a straight line"

Mathematically:

Linear function of one variable can be defined as:

$ f(x) = ax + b $

If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:

$ f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b $

If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:

$ f(x_1, x_2, ..., x_n) = w_1x_1 + w_2x_2 + ... + w_nx_n + b $

Adding more layer of linear functions doesn't make the function become "complex" for example, if you have $f(x)$ like below and then you put another layer of linear function $g(x)$ on top of it:

$f(x) = ax + b$

$g(x) = cf(x) + d = cax + cb + d$

as the neural network is trained to find the value of $a,b,c,d$, we can group the constant from the formula above, and then rewrite to:

$h(x) = mx + n$

with $m=ca$ and $n=cb+d$. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function

ReLU formula is a $f(x)=max(0,x)$, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ If you don't explain that picture, I think it's not clear at all why ReLU is non-linear. In fact, that picture says "rectifier" and doesn't even say ReLU and the rectifier looks linear. I think this answer can be improved! Maybe you should start by explaining what people mean by "linear" and "linear function". $\endgroup$ – nbro Mar 29 at 2:31
  • $\begingroup$ that's true, I will try, thank for the suggestion $\endgroup$ – malioboro Mar 29 at 2:32
2
$\begingroup$

ReLU is non-linear by definition

In calculus and related areas, a linear function is a function whose graph is a straight line, that is a polynomial function of degree one or zero.

Since the graph of the ReLU function $f(x) = \max(0,x)$ is not a straight line (equivalently, it cannot be expressed in the form $f(x) = mx + c$), by definition it is not linear.

ReLU is piecewise linear

ReLU is piecewise linear on the bounds $(-\inf,0]$ and $[0,\inf)$:

$$ f(x) = \max(0,x) = \begin{cases} 0 & x \le 0\\ x & x \gt 0\\ \end{cases} $$

But this is still non-linear on the entire domain:

enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.