Why should one ever use ReLU instead of PReLU?

Question

To me, it seems that PReLU is strictly better than ReLU. It does not have the dying ReLU problem, it allows negative values and it has trainable parameters (which are computationally negligible to adjust). Only if we want the network to output positive values it makes sense to use it in the output layer. Other than that, I don't see why a priori I would decide to choose ReLU over PReLU. However, most architectures I came across use ReLU activations. Why? Am I missing something?

Answer 1

I suppose, the situation is as follows - PReLU increases the expressiveness of a model for a bit at a small cost, but the gain is almost negligible as well (according to this post).

There is, indeed, a noticeable difference between ReLU and PReLU, since the former takes the same value for all $\mathbb{R}_{\leq 0}$.

However, compared with a LeakyReLU, note that this activation is accompanied with a linear operation, like Dense or Convolution layer: $$ y \sim f \left(\sum w \cdot x \right) $$ And the slope $\alpha$ can be absorbed in the weights of neural network.