# Is ReLU a non-linear activation function?

According to this blog post

The purpose of an activation function is to add some kind of non-linear property to the function

The sigmoid is typically used as an activation function of a unit of a neural network in order to introduce non-linearity.

Is ReLU a non-linear activation function? And why? If not, then why is it used as an activation function of neural networks?

Visually: if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia:

"In calculus and related areas, a linear function is a function whose graph is a straight line"

Mathematically:

Linear function of one variable can be defined as:

$$f(x) = ax + b$$

If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:

$$f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b$$

If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:

$$f(x_1, x_2, ..., x_n) = w_1x_1 + w_2x_2 + ... + w_nx_n + b$$

Adding more layer of linear functions doesn't make the function become "complex" for example, if you have $$f(x)$$ like below and then you put another layer of linear function $$g(x)$$ on top of it:

$$f(x) = ax + b$$

$$g(x) = cf(x) + d = cax + cb + d$$

as the neural network is trained to find the value of $$a,b,c,d$$, we can group the constant from the formula above, and then rewrite to:

$$h(x) = mx + n$$

with $$m=ca$$ and $$n=cb+d$$. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function

ReLU formula is a $$f(x)=max(0,x)$$, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.

• If you don't explain that picture, I think it's not clear at all why ReLU is non-linear. In fact, that picture says "rectifier" and doesn't even say ReLU and the rectifier looks linear. I think this answer can be improved! Maybe you should start by explaining what people mean by "linear" and "linear function". – nbro Mar 29 at 2:31
• that's true, I will try, thank for the suggestion – malioboro Mar 29 at 2:32

# ReLU is non-linear by definition

In calculus and related areas, a linear function is a function whose graph is a straight line, that is a polynomial function of degree one or zero.

Since the graph of the ReLU function $$f(x) = \max(0,x)$$ is not a straight line (equivalently, it cannot be expressed in the form $$f(x) = mx + c$$), by definition it is not linear.

# ReLU is piecewise linear

ReLU is piecewise linear on the bounds $$(-\inf,0]$$ and $$[0,\inf)$$:

$$f(x) = \max(0,x) = \begin{cases} 0 & x \le 0\\ x & x \gt 0\\ \end{cases}$$

But this is still non-linear on the entire domain: 