In an MLP with ReLU activation functions after each hidden layer (except the final),
Let's say the final layer should output positive and negative values.
With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.
However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative