# Is PReLU superfluous with respect to ReLU?

Why do people use the $$PReLU$$ activation?

$$PReLU[x] = ReLU[x] + ReLU[p*x]$$

with the parameter $$p$$ typically being a small negative number.

If a fully connected layer is followed by a at least two element $$ReLU$$ layer then the combined layers together are capable of emulating exactly the $$PReLU$$, so why is it necessary?

Am I missing something?

• @mshlis thank you for pretty printing – Robert Nowak Jul 23 at 15:02

Lets assume we have 3 Dense layers, where the activations are $$x^0 \rightarrow x^1 \rightarrow x^2$$, such that $$x^2 = \psi PReLU(x^1) + \gamma$$ and $$x^1 = PReLU(Ax^0 + b)$$

Now lets see what it would take to conform the PReLU into a ReLU

\begin{align*} PReLU(x^1) &= ReLU(x^1) + ReLU(p \odot x^1)\\ &= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\ &= ReLU(Ax^0+b) + ReLU((eye(p)A + eye(p)b)x^0)\\ &= ReLU(Ax^0+b) + ReLU(Qx^0+c) \quad s.t. \quad Q = eye(p)A, \ \ c = eye(p)b\\ &= [I, I]^T[ReLU(Ax^0+b), ReLU(Qx^0+c)]\\ \implies x^2 &= [\psi, \psi][ReLU(Ax^0+b), ReLU(Qx^0+c)]\\ &= V*ReLU(Sx^0 + d) \quad V=[\psi, \psi], \ \ S=[A, Q] \ \ d=[b, c] \end{align*}

So as you said it is possible to break the form of the intermiediary $$PReLU$$ into a pure $$ReLU$$ while keeping it as a linear model, but if you take a second look at the parameters of the model, the size increase drastically. The hidden units of S doubled meaning to keep $$x^2$$ the same size $$V$$ also doubles in size. So this means if you dont want to use the $$PReLU$$ you are learning double the parameters to achieve the same capability (granted it allows you to learn a wider span of functions as well), and if you enforce the constraints on $$V,S$$ set by the $$PReLU$$ the number of paramaters is the same but you are still using more memory and more operations!

I hope this example convinces you of the difference

• ok thanks, this sounds like a matter of efficiency and probably better learning/convergence capabilities. what does eye(p) stand for? – Robert Nowak Jul 23 at 15:40
• eye(p), is taking the vector p and making a diagonal matrix, where the elements of p is the diagonal (same functionality as like numpys np.eye). – mshlis Jul 23 at 15:48

Here are 3 reasons I can think of:

• Space - As @mshlis pointed out, size. To approximate a PReLu you require more than 1 ReLu. Even without formal proof one can easily see that PReLu is 2 adjustable (parameterizable) linear functions within 2 different ranges joined together, while ReLu is just a single adjustable (parameterizable) linear function within half that range, so you require minimum 2 ReLu's to approximate a PReLU. And thus space complexity increases and you require more space to store parameters

• Time - This increase in number of ReLu directly affects training time, here is a question on the time complexity of training a Neural Network, you can check out and work out the necessary mathematical details for time increment for a 2x Neural Network size.