Why do people use the $PReLU$ activation?
$PReLU[x] = ReLU[x] + ReLU[p*x]$
with the parameter $p$ typically being a small negative number.
If a fully connected layer is followed by a at least two element $ReLU$ layer then the combined layers together are capable of emulating exactly the $PReLU$, so why is it necessary?
Am I missing something?
Lets assume we have 3 Dense layers, where the activations are $x^0 \rightarrow x^1 \rightarrow x^2$, such that $x^2 = \psi PReLU(x^1) + \gamma$ and $x^1 = PReLU(Ax^0 + b)$
Now lets see what it would take to conform the PReLU into a ReLU
$\begin{align*} PReLU(x^1) &= ReLU(x^1) + ReLU(p \odot x^1)\\ &= ReLU(Ax^0+b) + ReLU(p\odot(Ax^0+b))\\ &= ReLU(Ax^0+b) + ReLU((eye(p)A + eye(p)b)x^0)\\ &= ReLU(Ax^0+b) + ReLU(Qx^0+c) \quad s.t. \quad Q = eye(p)A, \ \ c = eye(p)b\\ &= [I, I]^T[ReLU(Ax^0+b), ReLU(Qx^0+c)]\\ \implies x^2 &= [\psi, \psi][ReLU(Ax^0+b), ReLU(Qx^0+c)]\\ &= V*ReLU(Sx^0 + d) \quad V=[\psi, \psi], \ \ S=[A, Q] \ \ d=[b, c] \end{align*}$
So as you said it is possible to break the form of the intermiediary $PReLU$ into a pure $ReLU$ while keeping it as a linear model, but if you take a second look at the parameters of the model, the size increase drastically. The hidden units of S doubled meaning to keep $x^2$ the same size $V$ also doubles in size. So this means if you dont want to use the $PReLU$ you are learning double the parameters to achieve the same capability (granted it allows you to learn a wider span of functions as well), and if you enforce the constraints on $V,S$ set by the $PReLU$ the number of paramaters is the same but you are still using more memory and more operations!
I hope this example convinces you of the difference
Here are 3 reasons I can think of:
Space - As @mshlis pointed out, size. To approximate a PReLu you require more than 1 ReLu. Even without formal proof one can easily see that PReLu is 2 adjustable (parameterizable) linear functions within 2 different ranges joined together, while ReLu is just a single adjustable (parameterizable) linear function within half that range, so you require minimum 2 ReLu's to approximate a PReLU. And thus space complexity increases and you require more space to store parameters
Time - This increase in number of ReLu directly affects training time, here is a question on the time complexity of training a Neural Network, you can check out and work out the necessary mathematical details for time increment for a 2x Neural Network size.
Dead ReLu's - This is a problem in which a ReLu output becomes 0 due to negative input, so there is no way of flowing your gradient through it and thus, it further has no effect on the training. It can be made alive again only if the other alive ReLu's optimise some activations from earlier layers such that the dead ReLu again has a positive output. This is not very likely, since the loss is optimised by adjusting weights and not by adjusting whether dead ReLu's are present (basically it is not a learnable parameter so there is a random chance of it coming alive again, other ReLu's do not strive to make it alive). So, to accommodate dead ReLu's the size of Neural Net needs to be increased more, which again leads to added time complexity. Here is a question on Dead ReLU's. PReLu's do not suffer from this problem (which is probably one of the reasons of their introduction) and thus is definitely a better choice in terms of this criteria.
From personal experience, for a small number of epochs PReLu's tend to perform better than ReLu's for small number of epochs (I have trained only for small epochs). With further epochs and optimisation, this observation might cease to hold true.
